At risk of incurring disdain, I don't see anything wrong with that in itself.
We've got snow on the ground here and I'll be going out tomorrow wearing jeans, T-shirt and a hoodie - all of them cotton.
Apart from a few feet from my front door to my car and a few yards from my car to the church, I'll be indoors or in the car the whole time - warm and dry. If it's raining/snowing or looks like it might, I'll wear or carry my Goretex anorak. There's also a waterproof jacket in my car if conditions change while I'm out.

I really hate artificial fibres for everyday use and don't particularly like wool either. So cotton is my default choice for indoor/urban activities. I have a couple of pure wool sweaters for those times I need to spend long hours in an unheated building. Polyfleece is kept for outdoorsy stuff and emergencies.

I don't think wearing cotton clothes for a journey is a problem at all (assuming you're warm enough like that).
The problem comes with not being prepared for finding yourself NOT in the warm, dry situation expected.
Given that my car has a blanket, a shovel, maps, etc. and that I'll throw my winter journey pack*, a sleeping bag and my winter coat in the boot before I go anywhere, I think I'll be OK if things don't go quite as planned. I'll also throw in a pair of hiking boots (if I'm not wearing them), a pair of ice grips and a flask of hot water.

(* A daypack which includes extra food, a polyfleece jacket, extra hat/scarf/gloves and a bunch of other potentially useful stuff.
That stuff is in a rucksack rather than in my car so I can take it with me whichever car I'm in. It lives in the hallway along with a sleeping bag (in a stuff sack) and I grab them on my way out every time I'm traveling by car in winter conditions.)


Incidentally I don't have GPS in my car and am rather distrustful of it in my partner's car. There's a road atlas in each car and street maps of several counties in mine. We often ignore the GPS (we disagree with its directions on a regular basis), though we do use it quite a lot even on routes we know extremely well (to get an ETA or just to see how its ideas differ from ours). It seems to be most useful when finding our way to a particular place once we reach the relevant town and for when the in-laws phone and want to know how far away we are ("the car says we'll be with you at 20 past" seems to go down better than "we're on the A1234 and we'll see you soon") - the rest of the time it's just nice to have as a hands-free map on display.
When going to unfamiliar places, I plan routes ahead of time and print off maps, directions and alternative routes.