In one of Ray Meares programmes he recounts the story of a pilot who splashed down in Alaska. When he swam to shore Ray says 'his next decision saved his life' he took his clothes off, lit a fire and dried the clothes. Wet cotton is death. Note the order: wet cotton is many times faster cooling than even being naked.

Course the best option would be to wear clothes suitable for the climate you are flying over. Being in wet, but still warm, fleece that was rapidly drying itself would be no danger.
qjs