It helps a lot if you work to aggressively remove as much water as possible from fleece before putting it back on. Wringing doesn't work too well but spinning it over your head, letting centrifugal force remove water, and whacking it down hard on a flat, dry surface, both remove a considerable amount of water. Work it for a minute or two and you can get fleece remarkably dry. That and the exercise helps build heat.

You still will have some moisture left but far less than what you have with just a quick shake.