As long as you are not wearing accompanying camo face paint, probably wouldn't give you a second look. Of course, I've always lived in areas that had military bases close by.

I did hear that some place prohibit the wearing of Camo/BDU's off-station.