"I figured he was heading back to the office"