I would suggest heavily AGAINST messing around with reversing polarity on CR123s. There have been reports of those batteries exploding when stressed.

http://www.candlepowerforums.com/vb/forumdisplay.php?f=107

Some good info here on experiences. My own have been that there are pretty basic steps, but it should be ok to store items with batteries.

1. Don't mix a new and partially used CR123 - this seems to be the one most frequently associated with explosions.
2. Surefire packages their lights with batteries in the flashlight, and the tailcap unscrewed enough to prevent accidental activation. I can only assume this is a safe way to store them.
3. I would personally avoid reversing polarities, just to avoid any potential for messing with current flow. Some sort of barrier (paper, tape, etc) between the 2 batteries would be my preferred method, if #2 above wasn't used.