'When the leader’s office did venture into Defra territory, they didn’t talk to the shadow team...