"I just don't think that we would forgive ourselves"