I worry about a lot of things, but not about being a "gray man." I usually purchase gear and clothing on the basis of how it performs in the intended environment,not on how it looks. The profile the item might project is of little or no concern to me.

When going on a project or a trip, backpacking or whatever, i usually make it a point to include an article of clothing that is quite bright, if not indeed gaudy. I have spent way too much time looking for dudes who were said to be wearing faded blue jeans and a tan shirt. Still looking for some of them.

That said, there are times when you want to be inconspicuous, and times when you want to stand out. One of the best items I ever had was a reversible down jacket, loden (dull) green on one side and international orange on the other - worked great!