Personally, I don't think I'd off-road in sandals, but its your call. Maybe make a habit of wearing pants instead of shorts (or maybe the zip-off pants, and keep the legs in your bag) for a little better protection.
Good point. Is the area you're off-roading prone to snakes, scorpions, poison ivy, or general biting/stinging stuff? If so, boots may be a better option, and I'd definitely vote for long pants.