I think it's risky using partially depleted batteries as spares, unless it's just around the house, maybe.

Think about it. You are in a survival situation. Your batteries are dying. You switch to your spare batteries. Ooops - they turn out to have been more depleted than you thought.