Before the drop-dead date and within the operating temps of alkaline batteries, you really shouldn't notice any performance difference with alkaline batteries compared to lithiums -- except that the lithiums are lighter and have a higher power density.
I don't find this to be true in my case. My 4Sevens Quark 1xAA LED flashlight runs on standard alkaline AA's, NiMH (which I havent' tried yet), lithium primaries (also untried), or 14500 rechargeable lithium (which I HAVE tried - and use all the time now).
A 14500 is quite a bit brighter than the an alkaline in this flashlight. Actually, a TON brighter. No comparison really. Rechargeable lithiums have the highest voltage, followed by lithium primaries, followed by alkalines, followed by NiMH. Even though the NiMH have the lowest voltage, I still would expect them to give better brightness than the alkalines. I haven't tried this in this particular Quark flashlight, but in other lights I have the NiMH's will out perform the alkalines. Lower voltage, but significantly higher current capabilities.
Now in something where you don't have a "brightness" to compare, or in devices that only draw a low amount of current, like a GPS, I don't know how the various batteries will compare against each other. I generally use alkalines for these devices if they are used infrequently or NiMH if they see a lot of use. For intermediate use I choose the Eneloops (special NiMH formulation that doesn't self-dischage to a significant extent, like regular NiMH's do). I don't usually store the alkalines in these devices (due to leakage concerns), but occassionally I forget and run to open up the forgotten device and say "Phew, lucked out there - they didn't leak!"