"The Tories have taken up the cause because they think it can bring down my government."