For me, a good back up firearm would be a handgun in the same caliber. It won't do quite the same job as a rifle would, but at least you don't have to pack 2 different sizes of ammunition.
Obviously a larger rifle wouldn't have an easy counterpart in handgun, but if we're talking survival firearms anyway I wouldn't take a large caliber (situation depending of course).
$0.05