"Yeah, then you would've had to have a heartbreaking conversation"