I agree that you should layer your gear. Rather than having a single large backpack, take a duffel bag as your main gear bag (which stays with the ATV) and a smaller backpack such as a Camelback Rim Runner or Alpine Explorer as a walking kit. The walking kit should be light enough that you have no rational excuse to leave it behind; its primary job is to get you back to the ATV, even if that might be the next morning. Sounds like you had a good experience.