Hmm.... Every 23 I've seen is, at least as far as I can remember. But they are made of 8 stacked 1.5v mini coin cells, so you could do an alkaline. At least I think the energizer ones I mine for the LR44s are lithium.... Hmmm... I'll have to check.
I thought they were lithium.
I don't have any Energizers handy, but I have a pack of Duracells:
21/23
12V MN21/A23
They say alkaline right on the battery.
Looking on Energizer's web site for A23, I find the A23BP-2:
"Chemistry: Alkaline"
It also indicates 40 mah for those who are curious.
-john