Most of the camping I do is wilderness camping in the BWCAW. Since they have a "leave no trace" policy, I take everything I find. So far I haven't found much though. A length of rope here, a fishing lure there, nothing of significant value...
If in the BWCAW, and I can pack it out, whatever it is, I'm taking it out with me. Seems like the right thing to do there IMHO.