The tacticool gear is what caused me to remember one individual. He was a big guy, very casually dressed. That in and of itself wasn't memorable. The MaxP versapack bumping on his hip.... That's what did it. My first thought was he was carrying. Had he been carrying a non-tactical bag, I doubt I would have remembered him.
That's also a reflection of you and/or environement. I carry various types of bags. One is a MaxP. Another is a small leather bag that looks as non-tactical as possible without looking like a woman's purse.
In California, I am not unusual looking. I have been carrying a man bag for years, and I have asked various people. Everybody assumes I just have a lot of stuff to carry. However, when my family visits from Lousiana or Texas, they immediately ask if I'm packing. It doesn't matter what bag it is.
Anyway, going without a man bag is just super inconvenient for me at this point. I don't know how other guys do it, unless they enjoy wearing baggy, frumpy clothing with lots of pockets. I personally prefer my clothes to fit well on my body.