In terms of glass mirrors, a 2x3" mirror will be a lot less fragile than a 4x6" mirror. With plastic mirrors, a 4x6" will tend to warp a fair bit more. And, the more compact your mirror, the more likely you are to have it with you when you need it, i.e. a 4x6" isn't exactly pocketable.

We had been talking about aiming devices.

If you cant hit the target, it doesn't matter how big your mirror is. Even when painting an area, being able to accurately aim the device will increase the the probability of hitting a target in the area in a given amount of time (i.e. you will be more able to systematically scan the area). Same with scanning the horizon.