"We were better off before with Malcolm inside the Nation."