"But staying at the house every weekend, I just can't take it"