"If we kept this money, we'd be miserable for the rest of our lives."