Yep this is exactly right. Too many people are unaware that their votes are not anonymous on Lemmy and blocking the public tool only helps the bad guys who already know this. I've always thought this was a major weakness in Lemmy but I don't have a solution myself without some other major drawback.

I think probably votes should be anonymized or batched between servers so that only your instance's admins can see individual votes and you just have to trust the instances you federate with that they aren't pulling any shenanigans or otherwise defederate. That's not an easy problem to solve, but it's not like it's not currently possible to manipulate votes with a federated server, it would just be harder to detect. Regardless I think the need for privacy wins here.