There's an inner metal tank. Depending on whether it's gas or electric, there's either a flue running up the middle to the chimney(gas) or some electric elements inside. Of course there's also a cold inlet and hot outlet pipe. And there's an anode, which is a metal spike designed to attract any corrosive salts or chemicals in the water so they don't attack the other parts.

And, at the bottom of the tank, there is a layer of built-up sediment from the water and decaying anode. Some tanks have an access vent to clean out this sludge periodically. I don't know of anyone who actually does this, though. But the anode should be checked and replaced every couple of years.

See: http://home.howstuffworks.com/water-heater.htm