The trees I would allow to grow are called "crop trees", or desirable tree species which meet your forest management objectives. Often times oak, hickory, beech, and black walnut are considered desirable crop trees for many deer and turkey hunters.
Second to tree species is tree size, form, crown development, and canopy position. Ideally, we want these young "pop can-size" trees to become future canopy trees capable of producing a lot of hard mast for wildlife consumption and for future regeneration of your forest.
Depending on the acreage of land you have and your overall goals and objectives, it may behoove you to contact a professional consulting forester to assist you in your endeavors. You'll be surprised what a little professional forestry assistance out in the woods can do for you and for your prized forest land!