"I'm afraid this town will just go right back"