Not sure what your mirror set-up looks like, so possible I'm way off-base.

Is it possible that you might have overlooked an out-of-view set-screw or integrated tightening-nut (speaking from personal experience).

Or maybe a screw to tighten hidden within the housing (requiring removal of mirror assembly from door to access)?

Off-the-wall suggestions (since you want a repair that won't permanently fix mirror position); these would preserve ability to adjust mirror position:

Is there some sort of retaining ring that can be removed, allowing you to pull the ball from the socket? If so, how about cutting a tiny square (1/4-in, 3/8-in?) from sheet of medium or fine grit sandpaper, fitting into socket, grit-side towards ball (not so coarse that ball won't fit back into socket, but coarse enough to provide decent friction against ball surface).

Or -- assuming ball can be removed from socket -- insert a small square of very thin, compressible foam into the socket, then reinstall the ball.