"I go out of my way not to have an affair with this woman,"