1. The crowding and lack of personal space. NYC is the worst I've seen for this--I don't know how people can stand it. Makes it hard to get around and stresses me out being around a bunch of people constantly. Not to mention that if anything were to happen where you needed to get out of an area quickly, it would be impossible due to the crowds.
2. Traffic. Yuck. 'Nuff said.
3. Noise--you can't find any peace and quiet when you want it. Constant sensory overload.

I live in a town of about 21,000 with fairly easy access to rural and recreational areas as well as access to a city when I want to do something there. My commute is a ten-minute walk. I do live near a busy street which creates some noise, but all in all it's a pretty good arrangement.