"I had some fundamental differences of opinion with certain people in the White House."