From my time scuba diving, swimming in a wet suit is excessively tiring, especially on the upper body. This was in a 1/4" suit; I imagine swimming would be easier in a thinner rig.
Modern wetsuits are pretty easy to swim in, if they fit right. Considering the warmth and buoyancy advantages, I'll take them whenever I can get them, if the water isn't very warm.