I suppose the argument for a "grey man" approach presumes groups of people who could become a physical threat if provoked. So what provokes them?

A mob or mass protest is provoked by symbols.

Criminals are provoked by easy opportunity.

Other small groupos are provoked by territorial incursions.

The grey man approach, if there is such a thing, is to be uninteresting and unprovocative. Non-symbolic, not an obvious mark, and not looking to pick a fight. Just passing through.
But clothing or the way your gear looks is only window dressing; I think body language is the big divider between provoked and not.

But this all assumes contact. I like haertig's approach: the goal is to be elsewhere when bad things start to happen. This is an active goal; it's situational awareness in action.