Hello.
My setup is:
- Lenovo M920q mini pc with Proxmox installed (this doesn't have IPMI, only vPRO and it's annoying me)
- Fujitsu TX1320 M3 with TrueNAS Core installed - ZFS + RAID1 (this is a low-end "enterprise grade" server, and best thing - it has IPMI).
The Proxmox PC keeps all its CTs and 1 VM on the TrueNAS using iSCSI.
The idea behind my setup was that it felt nice that the TrueNAS would handle all the storage heavy lifting - ZFS, RAID etc., while the Proxmox mini PC would be a "compute-only" node that has a naked Proxmox install with some config.
The problem with that is if the TrueNAS machine loses power or is restarted, the Proxmox CTs/VMs switch their filesystem to read-only and stop responding to requests. This is because the iSCSI connection is interrupted. When the TrueNAS is back online, Proxmox doesn't make any attempt to restart the VMs/CTs - they'd still be broken.
It's annoying to me to have to VPN to the Proxmox web ui and wait 15 minutes until all the CTs/VMs are restarted and now again functioning on the "alive" iSCSI connection.
I was wondering what are my options here to remove the dependency chain?
I'm really into the idea of decomissioning the Proxmox node because I'm scared I won't be able to (over VPN) change the power state of the machine if something goes wrong, since it only has vPro and not iSCSI like the TrueNAS machine. By doing that, I'd consolidate the storage and the compute into the TrueNAS machine.
Options I can think of:
- Decomission the Proxmox node and move all Debian VMs/CTs to TrueNAS BSD jails. Is that even possible? Will all my Debian VMs work in BSD?
- Decomission the Proxmox node, switch TrueNAS Core to TrueNAS Scale and move CTs/VMs to TrueNAS Scale's Linux VMs
- Keep the Proxmox node and somehow figure out how to get Proxmox to refresh the CTs/VMs on iSCSI connection loss.
- Keep the Proxmox PC, but switch it to iESXI hoping that it handles the iSCSI failure more gracefully
EDIT: I didn't make it clear at first - TrueNAS stores more data than just VMs - documents, Linux ISOs (TM), photos, Syncthing
Hey, OP here again.
Here's what I ended up with:
upgrading my TrueNAS CORE to TrueNAS SCALE - it was really easy, just upload a 1.3GB upload file through the web UI. CORE's apps/plugins are based on BSD jails, where SCALE apps are based on Kubernetes/Docker, so I can any arbitrary Docker container from Dockerhub as I please, rather than being limited to BSD jails
migrating all the VMs/LXCs to matching TrueNAS SCALE Applications. So e.g. my hand-made Navidrome LXC was migrated to the TrueNAS SCALE Application. Sometimes there was no equivalent TrueNAS app for what I was using - e.g. Forgejo, so I just ran an arbitrary container from dockerhub.
decomissioning the Proxmox mini-pc (Lenovo M920q). I'll sell it later or maybe turn it into a pfSense router.
I installed a custom TrueNAS app repository called Truecharts. It has some apps that the default repo doesn't have, and it also has a nice integration with Ingress (Traefik), which allows you to easily create a reverse proxy using just the GUI.
I'm still yet to figure out how to set up Let's Encrypt for the services I made available to the Internet. I can no longer do things the Linux way, i must do it the Kubernetes way, so I'm kind of limited. Looks like HTTP01 challenges don't work yet and I'll have to use DNS01.
Looking back, I'm happy I consolidated. The hypervisor was idling all the time - so what's the point of having a second machine? Also, the only centralized machine has IPMI, so I have full remote control, and I'll hopefully never have to plug a VGA cable again. Of course, there's no iSCSI fault path anymore, though I'm happy I got to experiment with it.
The downside is as I said - I'm forced to do things the Kubernetes/Docker way, because that's what TrueNAS uses and that's the abstraction layer I'm working on. Docker containers are meant for running things, not for portability. I'm sad that I can't just pack things up in a nice LXC and drag it around wherever I please. Still, I don't thing I'll be switching from TrueNAS, so perhaps portability isn't that big of a deal.
I'm also sad that I ... no longer have a hypervisor. Sure, SCALE can do VMs, but perhaps keeping TrueNAS virtualized would give me the best of both worlds.