With a non-inflatable you would become buoyant inside the aircraft which could make it very difficult to move. Is your escape path totally dry just like it was before take-off? Probably not -- the aircraft interior may have a totally new configuration. The way out may be down.

Swimming well in the context of aircraft survival means being able to swim underwater, open a door/hatch that is under water and then go through that hatch. Doing that with an inflated PFD or non-inflatable PFD is not a good plan.

Finding a way out is your primary concern, once clear of the aircraft you inflate.