"You know, you're not the easiest man in the world to talk to."