Neutrals are what make arbitrary pairings work, which makes choosing them the highest-leverage decision in building a capsule wardrobe.
Two or three neutrals that work with each other is the target. Not every neutral combines with every other — cool greys and warm camels can fight, and a wardrobe split between them produces fewer outfits than either alone.
Build around your shoes and coat. Those are replaced least often and bought fewest of, so it is far cheaper to choose a palette around them than against them.