Yes keep both in your kit.
GPS units tell you where you are, compasses tell which way to go.
The flux gate compasses in GPS units are at best 5 degrees and only if you have calibrated them (every time you change batteries).
Practice with map and compass, check results with GPS.
Much can be said for the old compass, it's really a lot of benefit for not much cost. GPS's are great for position though.