I like mixing materials. My polyester next to skin layers consist of CoolMax or Nomex. Over that will be either a wool or poly fleece material and over that the opposite. For flight ops I have a nomex fleece jacket and I'm considering a nomex mid-layer. Nomex has some nice properties in the right weave. CoolMax, poly fleece and wool (inner to outer) works well. If it's raining add non-insulated raingear over the top.

Cotton is okay as long as it stays dry. At very cold temps when there is little humidity it can serve well as a tough abrasion resistant material that takes a spark from a campfire much better than nylon or polyester. However, once wet (falling into a river) it needs to come off immediately. Let it freeze and then knock out the ice. I remember many cold winters wearing jeans in the snow and I never got wet because the cold kept the water in a frozen state. Knock off the snow before you go inside.