Pretty much what Chris said. Personally, I think it depends on what you're trying to do, ie, car kit vs BOB vs some supplies for the house...

If for the house, things like food and water can be gotten at big bulk stores in large quantities (Costco, etc). Every case of something you buy, take one of the items out and toss it in your storage area. Look for deals at the supermarkets. Right now there's 5 cans of tuna for $4, but I'm pretty good on that (and probably will be sick of it by the time I've eaten it all). Don't neglect to include everyday items into your plans. Most guys have some tools - they don't have to be "special" or bought specifically for a disaster to be included in supplies. Same for clothes.

Car and BOB type kits... I had a good base of camping equipment to start with. Haunt online stores to look for good deals, or shop the folks under the "retailers" on the main ETS site. Places like Backcountry.com, Rei-outlet, Sierratradingpost.com have great deals at times, if you can wait for them to pop up. I would avoid places like Walmart unless absolutely necessary. And I personally abhor anything made by Coghlan's.

Don't buy everything at once. I did that for my girlfriend's car kit, and it was pricey. And like Chris said, don't be afraid to upgrade. Lots of my second-hand stuff I toss into the car, or into my gf's to upgrade what I bought her.