"There have been times when I lost patience with him..."