The way I understand the concept of "the gray man" is that he draws no attention, blending not with the natural environment, but the human environment.

What draws attention, and makes you stick in the memory of people, is color, contrast and movement. Even in a crowd of people that are brightly dressed, being not so brightly dressed can help you be a gray man. You do not want to contrast with the crowd, just not be very bright. In human environments, moving in ways that do not flow with the normal movement patterns also draws attention.

The point is to be forgettable. When someone asks: "Do you remember seeing a guy that looks like this", the preferred answer is no. They may have seen you, but you just did not make an impact on their memory.

That said, wearing clothing that is similar in appearance (style and color) will generally not arouse much attention by itself. There are "shiny" people to, people who just draw attention by their behavior and how they interact with others. If you are one of those, try not to interact with people any more than necessary.