Mitts for serious wind/cold will have a patch of sheepskin, wooly side out, on the back of the mitt. Mildly unpleasant to contemplate, but it's the reality of the human schnozz in cold weather.
Fuzzy velcro, anything cotton, or snow is the equivalent of sandpaper when frozen. Try it for a day if you don't believe me.
A piece of scrap polyester fleece could work, or soft wool if you have it, or a bit of a microfibre cleaning cloth.
<sniff, sniff>