When studies are done on juvenile mice, they use BPA concentrations that work out to be be about the equivalent of a 200 pound man eating a nalgene bottle a day. Being a fully grown adult with normal and healthy hormonal level, the amount of BPA that you get from normal (or even slightly abnormal but irregular) usage shouldn't have a significant impact.

It's like sacchrine. Sweet & Low carries a warning on it almost as dire as cigarettes, but we now know that the cancer risk for most people from it's usage is very low.