One of the problems I have with plastic containers is that eventually they impart a plastic taste to whatever is contained in them. I've used aqua-tainers and lexan and HDPE, and they all eventually do the same thing, especially if there's any sort of chemical already added to the water prior to storage (like bleach).

I figure using what is industry standard for consumable liquid storage is probably the most pallatable, though admittedly not the most convenient. 5 gallon glass carbuoys like the ones that are used for microbrewing beer seem to be a really good storage media. They are easy to disinfect, chemically neutral, and reasonably sturdy. They are heavier, and prone to breakage if dropped. Stainless steel containers do seem to work as well. All those bulk soda pop syrup containers seem to do the job without contaminating the highly corrosive contents. I wonder if those tanks are lined on the inside? I've never seen the anatomy of one.