Frenchy,
That's been my experience with bikes and backpacks too. It isn't bad if just wearing a daypack with a few items in it, but one time I tried riding a bike with my full sized backpack and nearly kissed pavement because my whole center of gravity was so high. Having loaded panniers too may make it easier by lowering the center of gravity. I suppose too that one could get used to anything with enough practice.