You have dialed in the answer pretty well. The drought of 2012 and a not so good summer of 2013 coupled with the severe winter have all contributed to the trees health. Leaf size is a good indicator of tree health. A thinned canopy could also be he result of bud and twig death caused by the cold winter temperatures. I am sure an arborist would suggest a plan to return the tree to better health would include feeding, possible canopy thinning to balance the root to shoot ratio and regular water on your part as a way to improve the situation.