"And I told him he can't just drop a challenge and leave."