To me, the worst part is they add a false sense of security. You may do something you wouldn't normally do if you are wearing one.
To me, JBMat's point about a false sense of security is also my own argument against your Joe Average prepper getting body armor--just in case. In general, that money is probably better spent elsewhere. Depending on the particular threats envisioned, tactics or other types of preps may ultimately be more effective protection than a vest. Or, put another way, if there were certain situations that I was fearful enough that I would want a vest, then there are likely other preps/planning I should do
first, and buying a vest would be one of the
last things to do.
Of course, when the rubber meets the road and a bullet is speeding towards your torso, of course I would prefer to be wearing a vest, but we usually have a lot more control over what happens before that moment comes than we think.