Well, as a former swimmer, wearing swim-type goggles for any significant length of time is gonna HURT! To keep a good seal they have to have a pretty good amount of tension on the retaining band. Of course, I would dive into the water just about everytime I used them so that does add a lot of pressure to the eyes, increasing the discomfort. Most of the models I'm familiar with also havve a pretty poor effect on your vision (loss of peripheral vision and tintng), which might adversely affect you if you are travelling long distances or over rough terrain.
If you are really concerned about eye protection (and it is a GOOD concern) consider close fitting sunglasses (I prefer Oakleys due to their suberb impact resistance and optical clarity) or professional goggles like those issued to the military (Wiley's, IIRC).
But even simple woodshop goggles would work in a pinch provided you had the ability to seal up any ventilation holes if things were getting through.