Just a guess, but I suspect that this phenomenon occurs because the skin temperature gets cold enough that the body reflexively starts to limit the amount of blood flowing to the skin (i.e. vasoconstriction). Unfortunately, the core of your body dissipates heat by sending the "hot" blood to your skin where it can be cooled down, so this vasoconstriction works against you when your body is hot. I think a product that moderately cools you (maybe 50-60 degrees?), as opposed to something that sticks a frozen chunk of ice or gel next to you, can avoid this vasoconstriction and also improve core cooling.

Izzy, how cold was the product you were using? Very cold?