I'm not an attorney, but I'm going to go out on a limb (ahem) and inform you that, "No, you don't have any rights to your neighbor's tree."
The story would be somewhat different if said tree's root system was clogging-up your septic system or said tree's limbs were extending onto your property and interfering w/ your quality of life and dropping smelly fruit directly onto your property.
Two potential suggestions:
1) Contact your city's forestry bureau to determine if there is a city ordinance regarding female ginkgo trees.
2) Contact an attorney
Best of luck!