I don't (yet) do kayaking, but I would consider a layered approach: Some basic items (fire starter, knife, space blanket or trash bag or whatever equivalent, granola bar, whistle, compass) always on your person. Not more than you can carry comfortably in your pockets, west or whatever arrangement you prefer.

You may be unable to retrieve your kayak and the kit in it, either because you've become separated or because you don't have the strength. You should still be able to huddle up somewhere without freezing to death, get a fire going and so on.

If you can wrap yourself in some plastic - ANY plastic - you've just eliminated heat loss due to evaporation, which is not a small thing when you're completely soaked in water. You've also improved heat loss due to convection, although not by very much.