"I can't drop by his house in the middle of the night"