Never bothered me in a decades of club level tournament bass fishng and recreational "pre-fishing." It is important to get an active sport -type pfd, like one for kayaking, and have it fitted to you. PFDs are not one size fits all.

Wearing something is way better than nothing.

And going fishing is too important to let something like a pfd choice get in your way. Good catching, OBG!