I don't think that's WHY it's stigmatised. It had to be stigmatized before that to be applied to selfish people.
It's a disorder that is harmful to others and difficult to understand. For others to cope with it requires navigating a complex network of negative behaviors. Manipulation and lack of empathy are the traits of a psychopath, and none of us want that.
The OP lacks empathy but evidentally desires it from others. If you understand why you need empathy, then you understand why lack of empathy is stigmatised.
This is the one. It's a perfect relaxation show.