Just the fact that the GPS tells you exactly where you are (in terms of coordinates) and where you are in relation to other waypoints is a tremendous benefit, regardless of whether sticking to a trail, street, sheet of ice, desert, lake, or ocean. A good knowledge of exactly where you are can be of great benefit when making decisions - where to stop for the night, which direction to go for help, etc...

Of course it your trail or streets have good signage or you have good reference landmarks, then you likely will be able to know you position without a GPS.

A knowledge of how to convert those coordinates to a particular place on a paper map - and vise vera - is very useful. Of course the map needs to have appropriate coordinates on it to be useful. This is discussed in detail at http://www.maptools.com .

I can't think of ANY location except maybe the dessert or oceans where people can move in straight lines without some kind most reasonable path to follow or obstacles to traverse.

As you say, a GPS - by itself - won't get you rescued, though it can provide information to help you make smarter decisions.