You could get the paper towel dispenser going before you washed up. That way yes you do contaminate by prepping and priming the paper towel dispenser, and you contaminate when you turn on the water and get soap, but once you are done rinsing, grab your preprepared paper towels, dry off, and use the towels to turn off the water and open the door.

Just don't let someone steal your paper towel before you are done.

Or carry some alcohol gel!