"I can't deny it, Lady Grantham would've made a powerful ally."