It is a matter of respect for others rather than of permission. When you show respect to others, even if you don't need permission, you are always taken more into account with the people in charge, who control everything that happens on the construction site. However, it can also become a security problem if there are people working.
On many occasions I have asked for permission to record and have been reticent and it is when I say
"I really don't need permission to record but if you dont, i will not record, it would really help me if you'd let me" or something similar depending on the situation and explaining why you want the images. When they understand that they can't really stop it, they put their "conditions"in place, I don't want my face to come out, or please don't record this and that way you get "permission" and there is a beneficial agreement for you that can provide you with a client in the long run. They still may not like the idea and it's best not to record. Why make people angry at you? Good publicity is better than bad publicity. The key? To be kind and informative and to advertise yourself