Because you get wet in those temperatures. If it's colder you stay dry. It is much easier to keep fight heat loss when your clothes are dry.

The increased heat loss in lower temperatures can (in principle) be negated with just adding another layer or two. Which is a whole lot of easier than fighting the effects of being wet.