"Worst thing I thought I was gonna be facing was working without Castle."