I wouldn't worry too much about heat and cold on standard ammunition loads. Two other things that temp extremes effect is your zero can move and lubricants change properties. Check your zero and use the lube sparingly.

Gun metal can get brittle down at liquid nitrogen temp levels. Unless you are travelling to the dark side of the moon, I wouldn't worry too much about shattering a cold gun by shooting it.