"The Conservatives can afford to lose..."