I bought my first pro-level kit (DW in 10/12/14/16/22 configuration) with this same idea: I could use whichever toms I felt fit the music better, and however many of them I needed. I have yet to come across a situation where I NEED more than a rack tom and a floor tom. Heck, I have yet to play a gig where my 12/14/20 setup wouldn't suffice. But, then again, I don't intend to play in a Rush or Dream Theatre cover band any time soon...You can spend the big bucks on a large kit when you have the money and just not use some drums to create a small kit.
You have to spend spend time finding other drums that match (if you like having matching drums) and money to get the drum, which may be a pain if you are financially hurting at that point and time.