If you are finding it unwearable maybe you should just take the sweat band out.
Newer Stetsons use a plastic material for the sweat band that does not stretch at all and is not really comfortable.
If yours is silk lined you will likely lose the lining cloth if you do this, but I found mine to be much more comfortable without the sweat band.

You might want to add a proper hatband, one that you can tighten, if it feels too large without the sweatband inside of it.
The hatband on mine was just a little strip of felt they stuck on to cover the line where they sewed the sweatband into the hat.
It was easily replaced with an adjustable band.

Mine is not exactly a dress or uniform item though.