"There's a family who's been at Downton for years. Very reckless."