"If he takes off, we'll never stick with him in this."