People tend to forget there's LOTS of calibers.
Not sure if it'll work for you, but .17HMR is a light round (smaller than .22) and works up to animals about 40-50 lbs (think large coyote). Maybe bigger with good shot placment. Bad side is the cost of the rounds is kind of hight.
I know there's also single-shot .223 rifles. It's a bit peppier than your standard .22, you can find ammo everywhere, and it'll take down deer size pretty well. Ammo is fairly inexpensive, and you can buy LOTS of surplus ammo.
Both are a lot louder than a 22 or a 38 special, tho the 223
is deer legal in California.
The NEF Handi rifles do come in a lot of calibers. I have
one with a 30-30, a 12 gauge and a 223 barrel. I have
carried the 12 gauge turkey hunting and carried the 30-30
barrel in my pack for pigs. Not a quick change over, it
takes one screw in the forearm, but if you saw game at a distance
and had time to stalk, it would work.
The 30-30 with peep sites weights 7 lbs, with the 12 gauge
barrel 5 lbs, with the lightweight 223, 5.5lbs.