I've done the same thing while scouting for trailheads. Sometimes they're overgrown with bush at the side of a road, and there are lots of game trails and false leads. I don't want to scramble through this scrub with full pack for nothing. So the pack stays in the car because I'll be back in ten minutes. In theory. Problem is, trails lead me forward, not back. One trail leads to another, and next thing I know I'm 20-30 minutes in with basically no gear. Not great.

Sounds like you've thought the matter through. An oversize fanny pack is the way to go. I'm currently testing a big one from MEC (6 litres of storage with external water bottle carriers). I got used to always having a handy pack like this while travelling -- it contained my essentials and my travel documents. I call it the "football" (IIRC that's the Secret Service nickname for the briefcase with the nuke launch codes).

BTW: I've only been badly turned around once, while bushwhacking. I would have bet any amount of money that my compass was wrong. I had to sit down, have a bit of water and a snack, and talk myself through it. These days, before a trip, I check my compass against a known north-south road. That way I won't question the compass in the bush.