I think the major problem is used cat litter, which can harbor cat scratch fever, which is usually harmless, but can be dangerous to infants. Unused cat litter is just bentonite clay, but clumping litter can contain silica dust, which is harmful to breath in. But silicosis is a pretty long term style disease. And just rinsing out a box should remove the dust.