I think of towels as being fluffy things too bulky to keep around except when needed. I do carry bandanas. I prefer silk to cotton because it's thinner and lighter for a given area, and equally absorbent for a given weight and thickness. 21" is about the smallest useful size, and will fit in a shirt breast pocket. 35" gives a lot more options and the diagonal is long enough to fit around any part of my body. Larger sizes get too unwieldy.

I recently bought a micro-fibre towel for its relatively small size, but I've not used it yet.