True. You're still depending on the other site to get it right, though. If the site makes a mistake, the password database may get copied, and then even if it only contains hashes, the hackers can use try to break them (for example, using "rainbow table" attacks.) They may not get your password, but they can get another string with the same hash that's as good as your password. They can also try dictionary attacks, as proved quite successful with the Gawker password leak (190,000 passwords decoded and published).

And that's if they are trying to get it right. XKCD makes this point better than I could.