"As long as he's a part of that family, there's no place for me."