"Look, half the time I'm there, he doesn't even want me there."