I think your philosophy about sampling to get the right sound for you is right on, it is a great luxury to be able to mix and match so many nice pieces. I have had a rather eccentric result based on the "mix and try" process but I can't think of what to give up at this point that won't compromise the result.
You can only rely on your own hearing and preferences, guided by a bit of theory.