"As soon as you let your wife and child leave New York,"