"Then at a time when she was in despair, we started an affair."