You need to compare what it would cost you to remove it now and what it would cost to repair the house. If you have any doubt, get a second opinion from a certified arborist. You may be able to have just the bad limbs removed. I vote for safety first!