"I mean, I can't be the guy that breaks his heart twice, right?"