There isn't going to be a great time to take it down from the chipmunk's perspective. The eastern chipmunk has two breeding seasons, one in spring and the other in late summer. Chipmunk gestation is 31 days with five to six young per litter. They are weaned at six weeks of age. This means you could already have young in the burrow. Once you see young ones coming out to forage, you might want to try to get the wall taken down then. Otherwise you'll be in for round two of this year's young. Potentially by the time the second group are out and about, it will be getting close to end of fall. What you are likely going to run into is trying to decide whether to evict young chipmunks during the summer months, or evicting the group right before winter sets in. Chipmunks hibernate over the winter. So it will be less stressful for them to relocate while it is still warm rather than trying to find a new burrow location as winter sets in.