Some say to replace the tires because the rubber compound starts to harden en you lose some of the grip provided by that soft rubber.
I suppose the question is: how deep does that hardening go? I have no hard evidence on that, though my tire guys say it is only skin deep -- as soon as you run on clear asphalt or concrete, the hard skin is worn off.
Some people want to squeeze the remaining life out of the winter tires by driving from 4mm to 1.6mm (legal limit) in summer.
I have done that on occasion. As you say, there is a cost in fuel efficiency. Handling is affected also, though not so much that a capable driver cannot compensate. It helps to increase the tire pressure to the maximum listed on the tire itself. Still, it is a compromise, and from a survival point of view, tires are a terrible place to compromise.