Besides irritating, it will severely restrict the range at which your signal mirror flash is visible, since your signal will be spread out into a broader, but less intense beam.

For maximum range, your signal mirror should give a reflected spot that is a tight circle whose diameter is about 6" plus the diameter of the mirror at 50 foot range from the mirror. I generally use a high quality glass signal mirror of the same dimensions as a comparative standard.

There's some discussion of this issue in this page:
http://babel.hathitrust.org/cgi/pt?id=mdp.39015010566720;start=1;page=root;seq=88

from this superb seven-page article by the father of the modern signal mirror, Richard S. Hunter:

Hunter, Richard S., "Heliographic Signaling Mirrors", Air Se Rescue Bulletin
U.S. Coast Guard, NAVCG 128 VOL. Ill NO. 2, February 1946.

The online version, alas, is clipped one one side by roughly 1/2 inch, but it starts here:
http://babel.hathitrust.org/cgi/pt?id=mdp.39015010566720;start=1;page=root;seq=86