"Oh, thanks, but we're really trying to enjoy this family time"