Not just your head.

The critical area's are your upper chest on a line from your armpits upwards (heart & lungs), neck, head and, oddly enough, your wrists.

Your neck and wrists are the points at which your major veins & arterys come closest to the surface. If they are not insulated then you can suffer drastic bodycore cooling and/or frostbite to your extremities.

The obverse is that you can remove hat's, gloves, neckgaiters and roll your sleeves up to prevent sweating.

For those who are interested this is called thermostating.