Okay, no one else said it so I will. The force of water is such that 5CFM can bend steel.(I learned that on the History channel the other day)
It is not outside of the realm of probability and well within the range of human experiences for water to literally rip things off that are attached to you.
If I were to carry a survival kit on my person when whitewater rafting, and I do, I carry it beneath the flotation device. There is little likelihood of it getting snagged, washed away or damaged. The vest offers some protection and since you would not need any of those items until after you hit dry land, their access or lack thereof is inconsequential.
The survival kit on a fighter plane is not accessible until after the jet jockey punches out for much the same reason.
I would have a knife on the outside of your PFD as well as a backup in your survival kit...