"Lady Merton and I were not exactly meant to be together."