"We all expected you to take the reins and that's not fair."