"I said, the colored man ought to be doing more"