"I made a comment jokingly that Caroline invited"