"After the grandchildren, it would've been hard"