"They'll annoy him into leaving his party"