GPS doesn't know what direction it is pointing when it is not moving, or is moving very slowly. Thus a supplemental compass might prove useful.
 
Not always true.  More modern GPSr have multi-axis compasses and give accurate direction even when motionless.  My Garmin 62s for example.  I do have to remember to recalibrate the compass after every battery change tho.  But no matter what I always keep at least one magnetic compass with me. 
I was referring to GPS as the satellite nav system, not the particular electronic gadget in one's hand. "GPS" has started to have varying meanings as onboard maps, onboard food & gas & whatever databases start to be delivered in the gadget. 
Some GPS gadgets have a flux-gate compass, some don't. Some have McDonald's restaurant locations, some don't.
Some even have cellphones attached to them!