I use the blue 7-gallon Reliance water jugs. They are cheap ($9 at Wally World) and fairly tough. The spigot cap is not perfect, but as long as you store them cap-up they are fine. They are not too big to lug around, and no pump is necessary.

I store them in the basement, up off the concrete floor and out of sunlight. They get rotated (in a perfect world) every six months. I just fill them from the garden hose and add a small splash of bleach to each jug, maybe a teaspoon. I make a point of drinking a quart or so of the old stuff every time I rotate it, and so far (5 years?) it has been fresh and tasty with no digestive issues. You can't smell the bleach (or I can't anyway) in such small quantities.

I've found that tying the rotation to other seasonal chores, such as draining the hot water heater in the spring and cleaning the gutters in the fall, helps me remember.