My expereince is that fleece is great in these kinds of scenarios. The fiber does not retain water, and can be easily wrung almost dry. If shielded from the wind by a plastic bag or a good windproof shell, you will warm up fairly quickly.

Build a fire or light your stove ASAP!