"But it's almost as if he's trying to avoid us"