No, two sets isn't anything to worry about. Sad, but not concerning. Strong winds or storms will knock baby birds from their nests. Sometimes cowbird babies will push the host babies from the nest. Sometimes babies fall when they are shuffling around each other jockeying for position to get fed or as they are trying to leave the nest, etc. Long-story short, it's tough being a baby bird, and lots of them don't make it. If you start seeing a lot of adults birds that have died, then that's cause for concern.