It's true and there is nothing to do, this situation always happens, like Sunwell and the same for others. And when it happens, few remain.
The rest goes to another server because the loss of interest in staying is mainly caused by seeing a few hundred players connected, despite the fact that several raids are still organized.
For some, the main reason to stay on a server is that it must have a few thousand players connected.
Just seeing so many players connected is the main motivation.
Others thinks differently.