The downside is (generally) more recoil and increased wear on the firearm. And if the bullet used is not designed for the higher velocities, then bullet failure can result as well.

Regards, Vince