If you are finding that a pack is throwing you off balance then the same weight will be much worse on the bike when riding off road.

A small, well fitting pack will work quite well as long as you do not overload it. It is the overloading part that often becomes difficult when you try to cover all possibilities such as spending the night out or dealing with major trauma.

As you have experience, mountain biking can involve major trauma and it happens often so I would lean towards more first aid items rather than overnight comfort if weight is an issue.