DJI recommends calibrating GPS / Compass only when the GO4 app says to. I've seen some guys swear they do it pre-flight every time.
MY P4P gets a little squirley now and then (drifts, lags taking commons, etc.) Upgraded to a newer iPad mini which seemed to help but still don't always have rock...