Every time people lament changes to the lore that amount to "not every member of species X is irredeemably evil" and claim the game is removing villains from it, I think how villains of so-caleld evil species fall into two cathegories: a) bland and boring and b)have something else, unrelated to their species going on for them, that makes them interesting.
DnD good and evil are distinct from common usage of the terms; they are cosmic forces, objective truths. Each action reverberates through the higher and lower planes and tilts the scales towards victory for one side or another in their eternal struggle. This lore doesn't leave a ton of room to change the alignment of entire races (and that is by design, structure makes it easier for people to get in to the setting).
But this is just in the established settings, any DM is free to homebrew any setting and justification they like.
Note that I am not trying to defend this as the height of storytelling, it isn't. It is a consequence of how the setting is justified - with deities being active participants, having specific rules for granting and revoking powers, and the physical presence of higher and lower planes embodying perfect conceptions of 'good', 'evil', order, chaos, etc. All of this can be changed, and again any DM is free to change it, but the 'deep lore' of the established settings over the past 40 years is drenched in this stuff.
One way to consider it is simply - the Drow aren't evil because they are Drow. The Drow are evil because their culture promotes actions that align with the literal true definition of evil that is present in the setting. Evil doesn't mean bad, it is just a label aligning with some physical rule of the universe. Just like the positive charge of a proton and negative charge of an electron are labels for physical rules of our own universe. Positive isn't any better than negative, but they are inherently distinct.
What you’re describing is closer to the nuance I’m interested in than WotC’s settings. If you read some of the later Lolth stuff, it’s the exact opposite of that. Evil is bad and the justification for anything always involves this trite movement from evil to good. They’re not presented as counterbalances or equal combatants. Even evil characters seem to always be working under the assumption that good characters are ultimately better.
The 40k universe has a lot of similarities. However, I’d argue its authors are somewhat better at presenting why Chaos is an equally valid choice or why the Orks can do whatever they want. There isn’t a clear choice (some authors are fucking terrible at this and drive WotC-style to the goodness of the Imperium).
The only reason WotC has to remove alignment from races is because WotC has made it very clear there is the thing people should want and there is the thing people should not want. That’s not an even layout of nine alignments. That’s a huge bias and all of their content reflects that.