OK, that helps clarify the situation. So you're thinking of the passport as a travel document and not just as a form of ID.
Actually, so you're sort of like Kris, at least when you're visiting your friends in far southern California, where there's a decent possibility of actually wanting to cross the border on a moment's notice, so in that case, I could see the utility of EDC'ing a passport in that situation since going back to your home to get it wouldn't be practical, whether for a fishing trip or during some emergency while you're down there. But if I lived, say, in San Diego (which I did) I never felt that EDC'ing a passport held any advantage for me over the risk of losing it or getting it stolen.
I would say that a passport in a BOB is a different situation from EDC'ing one, but that's a comment for a different thread.