"Lady Mary wouldn't allow it."