Wool is the answer, works even when wet.
I lived in Prescott for 15 years and am now up in Western Montana. I carry Ibex wool and a Golite raintop. If you're really worried a down coat in a stuff sack.