They don't leak like alkaline batteries!!! I hate the fact that I'm afraid to leave alkaline batteries in my rarely used devices. It drives me crazy when a nice device gets destroyed by leaking alkaline batteries.
I've switched everything that I can to lithium - especially devices that are difficult to open - such as electrical meters (multimeters) that require a screwdriver to access batteries.
Unfortunately some devices specifically warn against use of lithium batteries - for example, my Garmin GPSMAP 60CSX.
I too find myself liking CR123 batteries. Bought in bulk on-line they can be found at reasonable prices.