The difference is that the electrochemical reaction in a primary cell Is not meant to be reversible. If you recharge a primary cell, it won't work as well the second time around and is more prone to leakage.

I run Eneloops and protected 18650s in frequently-used devices. Otherwise it's Energizer Lithiums or Titanium Innovations CR123A lithium primaries. The one place that I still use alkalines is in my old Maglite 3D flashlights with LED upgrades. They're not very bright compared to my newer flashlights but I can drop them from dizzying heights onto concrete and only add slightly to the character they've accumulated. I've fished them out of the sump pit more than once without any ill-effects.