I consider, Silverfox's "Alkaline Battery Shoot Out" (includes AA and AAA lithium cells) the definitive word on alkaline battery performance. He also did one on NiMh , Li-ion , 123A cells and battery chargers.

Personally, I try to use lithium cells wherever possible. There are ways to make this more affordable.

1) Order the batteries online. You can get a much better deal through the Internet than in brick and mortar stores, esp. for 123A cells.
2) Some devices are more efficient than others. In a flashlight, get an LED light with multi output levels. The new LEDs are getting really impressive in terms of efficiency and it is amazing how much a multi-output light will save.

Next, when it comes to re-chargeable devices, I try to get ones that use lithium-ion batteries (read NewBie's "Li-Ion protection technology and possible dangers"!).

In fact, my EDC flashlight pretty much uses all the tricks listed above. It has four primary output modes .3 lumen, 10 lumens, 30 lumens and 85 lumens. It has an 18650 lithium-ion battery pack as my primary carry, with a 2x123A battery pack as my spare.

I do keep alkaline cells, but only for things that are not compatible with lithium batteries, or for things (as you point out) that get used quickly.

-john