My guess is that the "friends" telling you this have rarely if ever been in a real fight, and if they were it was mild or they were on the winning side. I have a hard time with this sort of peer pressure. Hide the can if you don't feel comfortable getting ribbed a little.
This is all about appearance and bravado, and who cares if it makes you feel safe? The guys who say this sort of thing are often the ones who need it most... mouthing off at a bar, flipping the bird in traffic, trying to show off and act tough.
A lot of the LEOs I know carry pepper spray, do you think they are girly? I'm fairly skilled in martial arts, but I would carry the stuff if it meant avoiding a fight. In this day and age, it's not uncommon for a fist fight to turn into a deadly brawl involving knives, guns, broken bottles etc. I could defend myself fairly well, but what if it's a few guys who are trying to jump me? That can of spray could even the odds. Same thing if I'm trying to protect my family... staying and fighting is almost always the wrong choice. If a fight is imminent, that could buy you some time, saving your life, a broken face, a ruptured eye socket, knife in the ribs, and any number of other things.
Tell your "friend" to mind his/her own business, or just find better friends.
_________________________
Experience is a hard teacher because she gives the test first, the lesson afterwards.