At least in theory, this is mitigated. The signal activation server sees your phone number, yes. If you use Signal, the threat model doesn't protect you from someone with privileged network or server access learning that you use Signal (just like someone with privileged network access can learn you use tor, or a vpn, etc).

But the signal servers do not get to see the content of your group messages, nor the metadata about your groups and contacts. Sealed sender keeps that private: https://signal.org/blog/sealed-sender/

You would obviously want to join those groups with a user Id rather than your phone number, or a malicious member could out you. It's not the best truly anonymous chat platform, but protection from your specific threat model is thought through.

edit: be sure to go to Settings > Privacy > Phone Number. By default anyone who already has your phone number can see you use signal (used for contact discovery, this makes sense to me for all typical uses of Signal), and in a separate setting, contacts and groups can see your phone number. You will absolutely want to un-check that one if you follow my suggestion above.