When you're paddling a kayak or an inflatable you're normally wearing a PFD so that's some extra help, plus you're spending most of your time sitting in a boat rather than immersed.
For sustained swimming, which I rarely did, I think warmer would be better.