These filled 5 gallon water bottles run about $7.00 here, if you return your old empty one when you buy the new one. If you don't return the empty, a new one costs about $14. You can find them in the grocery stores, office supply stores, home improvement stores, etc.
Sooo ... you can "buy" one of these nice 5 gallon water bottles to keep (just don't return it) for $7 if you want. Not collapsible, but built sturdily. And they have a handle to carry with and a nice neck for pouring (although there really didn't intend them to be poured I don't thing). Once you've removed the cap, you can smack it back in place later, and it seals up nicely upon re-use (the lower part of the cap is torn off and thrown away however). I don't know how many times you could re-use the cap, since we throw ours away 95% of the time. You could fashion a new "cap" with Saran Wrap and a rubber band though. You can also get a 3 gallon version for slightly less money, if you can't manage the larger bottle.
We have these delivered (for a small additional cost), and we always keep around 6 bottles in queue in addition to the one that sits in the water chiller/dispenser. We cycle through the queue, first in first out, so the water we have on hand is "fresh" (no more than "6 bottles old" - the size of our queue).