I never would have thought that "gray man" meant that you had to wear gray. I always interpreted it as a concept that meant "don't stand out". So if the crowd is all wearing fluorescent orange, then that is a good choice for you to wear as well. "Gray man" is a figurative term, not a literal one, that really shouldn't require an explanation.