If the squirrels come within a few feet you may be able to see the anatomy of the males. Females may be a little easier to pick out if they are lactating. The coloring is the same for both genders in both gray and fox squirrels. And size is typically a matter of age (juveniles are smaller) and the abundance of food resources.