If so, is the "expiration" only for taste purposes?
Unlike tap water, bottled water is relatively lightly regulated. I'm not sure if there is even a uniform definition of "expiration date" when it comes to bottled water.
I think it's safe to drink bottled water well beyond the expiration date. And, if in doubt, just purify it when you use it.
One point that others have raised in the past based on their experience is the durability of the containers long-term. The soft, HDPE jugs don't seem to hold up well over long periods of time, developing pinhole leaks. PET bottles will probably do better than HDPE, but for long-term durability and avoiding surprise leaks, a more heavy-duty water containter may be preferable.