"I don't know if it bothered him that I stopped coming around,"