Keeping people on board might be about passenger safety as much as anything. Or it might be a delay for customs formalities.

I imagine that the captain of the ship has this authority based on the jurisdiction the ship is registered in. I assume this is spelled out in the legal documents that passengers sign.

Depending on the situation, police from the local jurisdiction could conceivably be stationed on the land end of the gangplank.

If someone wants to jump overboard and swim to shore, no-one could stop them. But they would then have to work through the customs/immigration authorities in that jurisdiction. Potentially messy.

But this is hypothetical. It sounds like passengers were taking day trips on the island. Hardly a floating gulag.