Like the OP, I carry it in my wallet. Emergency contact information (ID is my DL), perscription drugs, etc. But, the information on it is by no means proof of identity. It's just a hand written card, and there's not much that can be done with it. The rest of the cards in my wallet... Yeah, I might freak about those. The increase in risk of the carrying a medical or ICE card is so low compared to the existing risk from the rest of the contents of my wallet, it's just not worrying about.

Now, if it was a life threatening allergy or medical condition, I would be wearing a medical alert bracelet or pendant. I wouldn't risk a medical mistake during the delay between scraping me up and checking my wallet for ID and insurance.