Consider the surface area of your head is the same as your upper back or chest (remember the rule of 9's). You wouldn't go out in 20 degree weather with your upper back exposed, would you?