They have been sanctioning sites like pixiv for years, and last year also were the reason Ken Akamatsu's (Love Hina author) manga archive site for out of print manga was shutdown, and why sites like Melonbooks (one of the largest in japan) only can accept JCB payment now.
Visa and MasterCard have a history of sanctioning anything that has anime and manga adult content, and I'm not talking about loli, just any adult content.
You don't have to care.
No matter what you do, someone will think poorly of your actions. You can't please everyone.
Prioritise the things you personally care strongly about and leave the rest to the people who care strongly about those things. If that makes you a bad person, then maybe embrace that you're not perfect, and neither is whoever criticizes you for it.