They had a permit to act as a test / lapping track and proving ground which is what they used it for. Most of the noise that the locals complained about was from the Indy Lights / Cup Light Cars and the race teams that used the track for tests. This is how the track made it’s money; not from our lapping days.
The only thing they could not do was have wheel-to-wheel racing and that type of competition which would have required a different permit. It wasn’t classed as a “race” track and never used as one.
The locals tried to get the area re-zoned at one point to make the permit they had invalid. But the permits are based on how the track is used, not on what kind of cars use the track. The track also tried to get the area re-zoned in their favor. Neither effort was successful.