i take fleece,or the lighter stuff i call poly fluff,on all my canoe trips and what i found that really keeps me warm is to take off my shirt,put the poly fluff hoodie on over bare skin and then put the shirt back on.
I've also tried that and can highly recommend it.
BTW, this is one place where the wrong kind of wool comes in second place to synthetics, IMO - contact directly with the skin. The wrong kind of wool can be itchy and drafty. I recently added a thin wool jacket for an emergency extra layer in my urban EDC kit, but it's with the mind set that I likely won't get soaked to my base layer. In a wilderness situation, I take 2 extra layers - a wool and a synthetic.