Selfhosted

If you want to get fancy: systemd credentials. It can store the secrets encrypted on disk and seal the encryption key with the TPM chip. The encrypted secret is decrypted (non-interactively) and made available only to a specific systemd service. The process itself does not special systemd integration--it just sees a plain text file containing the secret, backed by a tmpfs that's not visible to other processes.

Depending on which TPM PCRs you bind to, you can choose how secure you want it to be. A reasonable/usable configuration would be something like binding to PCRs 7 and 14. With that setup, the TPM will not unseal the key if the system is booted into any other OS (i.e. anything signed with a different UEFI Secure Boot key). But if you really want to lock things down, you can bind to additional PCRs and make it so changing any hardware, boot order, BIOS setting, etc. will prevent the TPM from unsealing the key.

Yep, this is a root of trust problem. Your choice will ultimately come down to how much you want to invest and how much inconvenience you'll put up with, measured against how secure you want it to be.

Personally, I go for full disk encryption and then just store things on the filesystem in secure (to the OS) ways. File permissions and users and groups, etc. Most other things boil down to that though something like vault adds a layer of access control in that you can seal it off in the case of a breach (if you care) and can get granular with authz permissions in a centralized place, only managing authn in your distributed tools.

There's probably some ideal system out there like vault but with a plugin that can ping your phone for quick verifications that would likely be ultra ideal, but I haven't seen that. Personally I'd love something like that.

Environment variables. If they're in my network, that has no open ports to the internet, I've got plenty more problems.

Even a dev machine, think about how many env vars a normal dev has, plenty to loose.

Secrets management tool for self hosting on my level would bring more complexity for little gain.

Bash scripts etc can be uploaded to open repo and not share secrets which is what I want.

Hashicorp Vault + Vault Config Operator + external-secrets. I have a simple chart that can add credentials to different apps which mostly gets used in argocd with its multichart functionality. A simple bash script to create the vault policies, which use the k8s back end to allow auth.

Ideally, a secrets manager that you can unlock once and then give access to the secrets. You can either unlock it at boot by entering the password, or if you have a TPM, you can also do something to encrypt the main key with the TPM and then when you boot up, if all secure boot checks passes, you can decrypt that key with the TPM and unlock the secrets manager in question.

You can also store the secrets on another machine that only exposes say, the Vault API. Like a dedicated Raspberry Pi just for this function where remote access is disabled and everything, it only serves secrets. That way, you can trust the Vault logs to know when each secret was accessed and find anomalies this way.

If someone breaks in via SSH or whatever, your security already fell apart. That box can no longer be trusted for anything. Especially if they breached the root account. Doesn't matter how big the fortress is, if you're inside, it's game over, it's time to evaluate the damages and clean up.

I use shh keys for all my remote machines, set passwords automatically with ansible, and store them with pass.

https://www.passwordstore.org/

EDIT:

Just to clarify, ansible can use pass as a password store, so in the ansible playbooks you can write which password you want to retrieve from pass.

You can also call pass from any shell script by writing $(pass <target_password>)

I generate a unique key pair (or token) for each service that I want to access from the host machine. I see no issue with storing that dedicated private key locally in plaintext (obviously in a folder where only the required user can read it and I except it from backup and versioning). I use one dedicated user per externally accessible service.

Should the machine itself become compromised this would indicate that my personal master key and master password have been compromised or someone gained physical access. That would require me to restart from a blank page anyways.

Yeah, I haven't gotten past using plaintext secrets in separate files (eg. using Home Assistant's secrets files). I do the same for any Python scripts that need secrets too, like Slack auth tokens and stuff.

I haven't really gotten around to looking into secrets management in my homelab, and I know I really should. As much as I have a lot of faith in my Nginx and Authelia config, it only takes one hole for someone to get in and get to those secrets files, especially if that hole is a security flaw in something like Home Assistant - one of the very few services I can access publicly, without using my Wireguard VPN.

• For kuberentes stuff, I use external-secrets-operator. It can be paired with selfhosted stuff, like Hashicorp Vault.
• For machines, disk encryption LUKS, and then files in plain text, with minimal permissions: mode 400.
• For my laptop, keepass.

In my opinion, for home selfhosted stuff you don't have to go for complex solutions. In the industry, the problem is that secrets needs to be served to different systems, by different people, with some kind of audit logs. Unless you are working with lots of people, environment variables are OK. You github/gitlab may have all scripts with variables, and your disk may have a .env file with mode 400. If you make any machine or container with a single responsibility, there should be no secret leaks among them.

For example, let say your wordpress instance gets pwned. It should only have its needed secrets (like its db credentials), so your wikimedia instance is still fine.

I haven't played around with it much, so this is a pretty uninformed suggestion, but BitWarden have a secrets manager, currently in beta. You can create projects, then grant human or service accounts access to secrets. Looks promising, but I have to say I'm a bit of a BW fanboy so I concede some bias.

Not ideal but a few scattered ssh agents I unlock on reboot. And zabbix to tell me if they’re not running.

Two more options you might consider:

• secret-tool - like a vault that unlocks when a user logs in to their session. This shifts the problem to keeping the user's login credentials secure but depending on your setup that might be preferable. Just be aware the once unlocked any process could access the vault in theory (I wish they'd add access controls...)
• podman secrets - so you can securely provide secrets to containers. You can set these once securely then nothing except processes in the container can get them.

I'm on Kubernetes with ArgoCD gitops.

I use sealedsecrets, so all my secrets are in git encrypted. The encryption key is in my keepass vault.

Is cert based auth an option?

Ansible vault. All my config files and scripts are deployed with Ansible. Usually they are pushing those into a file or environment variable but if you scope permissions narrowly and don't run services/containers as root you should be somewhat safe. If someone has filesystem access you're already in big trouble.

Instead I'd focus on keeping your attack surface as small as possible. Keep services behind a VPN or segment public facing services to a separate VLAN or docker network.