So do I. My cold weather mittens are a pair of thick fingerless wool gloves with a windproof layer on top. (Some cotton canvas or synthetic thing, semi-water repellant). I use these when temps drop a few degrees below freezing or colder.

These kind of mittens are standard for equipment for cold weather skiing. Comes in all price ranges.

Operating brakes and gears aren't exactly "fingerspitsgefühl", but good enough.