If you're talking a house evacuation where you're not hurt or in need of emergency medical care, scrubs might work. They don't stretch at all though, and I personally wouldn't find them comfortable to sleep in for that reason. But what happens if you're in the shower when your house catches on fire?

But if you're worried about it, why don't you just put together a "scramble bag". Put in a full set of clothes and shoes, towel, hair brush, toothbrush, and whatever else you think you might want to be presentable, or to live in a neighbor's house for a day. So you could run out of the house naked if need be, then find a tree to hide behind to get dressed. Just keep the bag stored next to the front door or in some central location that you'd have to pass by on your way out. These kind of bags are quite common - I can't tell you how many of these I picked up for pregnant ladies going into labor when they called me to respond with the ambulance. Not talking about a full-on Bug Out Bag, just enough stuff so you could make it to the store to buy more as needed.