It is my understanding the only way an airport controller can deny a flight is if the airport is closed to all traffic or for special evens. If I have authorization, and the airport isn't closed, I'm going to notify the airport manager, normally that means leave him a message and perform my flight. The only exception would be is if my flight was in direct line with approach or departure and then most likely one wouldn't get authorization. I would file a NOTAM.
I think more ATC guys are being brought up to speed on FAA regs regarding drone operations, remember, the ATC guy is just suppose to control traffic, not make up rules. Now if all flight operations are canceled for whatever reason then that means drone flights are canceled. If the airport is IFR only due to visibility or surface ceiling then drone operations could be denied.
ATC can deny a flight if they have a legit reason. Your COA is in theory "if everything is perfect". Take for instance last year when I was flying with a CFII (I was PIC actually in the aircraft ) and we were flying in an area adjacent to the runway. We were doing some practice approaches etc and all of a sudden 2 military transports came in and they decided they wanted to shoot some practice "Cargo Drops" in the area. Guess who was the low guy on the totem pole that day? Yep the Cessna 172 got shooed off to a practice area 7 miles away?
So if they have a legit reason our pre-approval isn't going to mean anything. Now if your authorization/COA doesn't stipulate to call the tower then they have no way of knowing you're there nor can they deny the flight.