"But the fact is you let him go up that ladder and he died."