Greetings Charles:
Well, the short answer is, "yes, removing half the branches from this walnut tree will have an impact on the health and vigor of this tree."
Beyond the obvious, this is a legal question that should be remedied by both parties. Technically, an adjacent homeowner can cut plant branches that extend into his/her yard --if and only if -- the cut branches do not cause adverse effects as related to the health, vigor, and longevity of the tree in question.