As I indicated, in warmer water you last longer. I seem to recall reading that in WWII in the S Pacific, some sailors floated in lifejackets for 24 hours or longer, and survived. In that case, as Nursemike indicated, sharks and other marine life become bigger issues. Not to mention sunburn, dehydration, skin damage due to prolonged water immersion, etc etc.

Any estimate of survival time in water (cold or otherwise) is just an educated guess. In Alaska the USCG routinely searches for people in the water a good deal longer than the charts would suggest you could survive. They like to give you the benefit of the doubt.