100 years ago in my first year law school, in our "property" class we discussed, the traditional property law of "you owned everything above your property to the heavens" as modified by commercial aviation. At what altitude, did a property owners rights end. So, nations do not have "property rights" to deny satellites flying over their territory (as long as they are in "space") and individual property owners do not have the right to deny aircraft the right to fly over their property, again, usually assuming some minimum altitude.
Drones are requiring new laws to be made. I can deny a 6 foot person from my property under "trespass" law. Can I deny a drone flying at 6 feet? It does not seem to be clear. . .
Here's my opinion about at least some of it:
I cannot speak of the legal ability to shoot one down, but at the very least there should be reason to involve police and even (or especially) a lawyer: suspicion of photography or video recording.
In at least some jurisdictions, it is illegal to photograph (and for the sake of the rest, it will include video recording) anything which cannot be seen from a public space -- and presumable from your own property -- without the use of special equipment.
Special equipment includes things from special sensors, such as IR or UV, to see beyond barriers, to ladders (and climbing a tree in at least one case I read about), and has arguably meant telephoto lenses. In my opinion, using a drone to photograph private property is absolutely use of special equipment.
If someone spots a quadcopter being flown above their property, especially considering there are plenty models (cheap or otherwise) with built-in cameras, the suspicion should be that it is being used to record footage.