My response was in reply to
@MapMaker53 comment about needing a law.
I agree with your comment. If someone gets it in their mind to confront you in a hostile manner, it may be difficult to talk them down. As I said above, if they threatened to call LE, I would agree with them that they should do it as soon as possible. In most cases that leaves them standing there with a blank look on their face followed by their retreat while mumbling something.
I think it is rare that people will go out of their way to confront you. If there are people near by, they may comment one way or another. But to go out of their way and approach in order to voice their displeasure is uncommon from my experience. I have been approached in that way a couple of times. I think it's inevitable if you are fly enough. I think all we can do is keep a cool head and be polite as possible while at the same time letting them know that there are laws in place and that we know what they are and have an FAA certification to prove it. Then ask them if they do. Of course they don't and it gives them something to think about.
Being apologetic and quickly loading up your gear and getting out of there would leave the with the wrong impression.