I'm pretty sure that an N95 is overkill. A regular dust mask should be fine - you're filtering pollens/dusts/allergens, not bacteria. After all, no one wears an N95 when they're doing yard work... do they?
I do. I'm increasingly bothered by the mold spores in composters and dead vegetation -- gives me mild flu-like symptoms the next day. Dust masks don't seem to help at all, but an N95 gives me a bit of protection because of the filter media and the two strap system.
Note that fit, fit, fit is critical to their effectiveness. Standard N95s don't fit me all that well -- I don't get a great seal. The (stupidly expensive) ones with the exhaust vent are much better. (I'm experimenting with stick-on weatherstripping to improve the standard N95's seal. We'll see)
Off-topic but interesting: during the SARS epidemic a few years back, surgical masks were found to be inadequate to stop the airborne spread of the virus. N95 masks were the minimum requirement.