That's a good question. I think if one were to wash thoroughly and allow one's hands to dry completely, that one would be better off than not washing at all, but I don't have anything to back up my guess. I suppose it might vary with type of contamination.
However, I don't normally encounter that situation. When I'm out back packing, I normally wash with treated water. There are also various types of hand sanitizers and baby wipes out there, although soap and water are considered to be the best, and soap and water don't develop antibacterial resistant bacteria.
One item of note: according to Discover magazine, one should scrub one's hand after soaping up for at least 30 seconds before rinsing off. Some friends of mine in the food prep industry say that washing twice at 30 seconds is what they've been taught.