The main reason I don't like cars and small SUV's is they are all unibody. What happens if you are evacuating and someone comes too close when they cut you off and you get in a minor fender bender? With a unibody vehilce your now dead in the water but in a frame based vehilce you can keep going and fix the bodyt damage later.
Fuel capacity is another issue. Smaller cars and SUV's have smaller gas tanks so even if you can get better milage you don't have as much range. Then sitting in traffic the small cars don't get very good milage either.
Also towing, a car like your camry can't safely tow. safely is the key word here, I'm seen people tow with front wheel drive cars and I always keep faw away from them.