Remember, there are two (well three I guess, if you count friction) things that will cause load on the compressor (and slow down the spooling). The air being compressed, and the rotational inertia of the compressor wheel. Being lighter, and having more of the mass further inwards will make it spin up more easily due to lower rotational inertia. Once the air itself really starts to place a load on the compressor, you’re at the point where its doing the work you want it to.