As a rule, it's best that new clothes are washed if they will have a lot of skin contact. Some of the dye residues and other chemicals are probably not all that good for you if absorbed, or may cause nasty skin irritation. And keep in mind that offshore manufacturing is a wild west in terms of chemical safety.

The exception to the rule: guys mostly don't bother to wash stuff. Whether this is good or bad is up to you.