I've considered using a pellet gun as a second gun. I think it's a good option, mostly because so much ammo can be carried so easily. I think I'd sacrifice some muzzle velocity and go with a handgun version rather than lug around a second rifle.

Right now my second gun choice would be either a S&W Kit Gun or Charter Arms Target Pathfinder. The latter can be switched between .22 lr and .22 mag by swapping cylinders, which seems useful. It's hard to find CA guns in the shops, so it might be a while before I get my hands on one so I can compare it to the Kit Gun.

If I lived in an area where hunting small game with a handgun were illegal, I'd probably buy a take-down .22 like the ones put out by Marlin or Henry Arms. The lightest one I know about is the Pack-Rifle, though I've never shot one. It looks cheap: http://pakrifle.com/Pak-Rifle.html

One more idea is to use a large-caliber, 5' blowgun as a "second gun". It can double as a walking staff. Ammo is pretty easy to make, and I know it's good for squirrels, at least.