Not necessarily. Even if it's just a lump of sand, it's possible to survive on just rainwater and fish, and maybe even some birds. You can't eat or drink the fuel, but you can use it to boil seawater for rudimentary desalination, and to cook the fish. I don't know much about fishing, but maybe the fire could also be used to attract fish at night.
And that signal fire becomes all the more important in such a situation. Even with no food or water, you may survive long enough to be rescued if you're very lucky.