My apologies for the long post.
I have a single server running Unraid with about 12 services (Pihole, Wordpress, Heimdall, Jellyfin, etc.) all running on Docker. This server is also acting as my home lab NAS. Everything runs fine for my use case (at least for right now) but I’ve been thinking about creating some type of compute cluster for my services instead of a single server.
Recently, I saw a discussion about Proxmox, Docker, LXD and Incus where a user felt that Incus was a better option to all the others. Curious, I started reading up on Incus and playing around with it and contemplating switching all my services from Docker in Unraid to an Incus cluster (I’m thinking around 3 nodes) and leaving the Unraid server to serve as a NAS only.
In a nutshell Incus/LXD appear to be (to me) a lightweight version of a VM in which case I would have to manually install and configure each service I have running. Based on the services I have running, that seems like a ton of work to switch to Incus when I could just do 3 physical servers (Debian) in docker swarm mode or a Proxmox cluster with 3 Debian VMs with docker in swarm mode. I’d all possible, I would like to keep my services containerized rather then actual VMs.
What has me thinking that a switch to Incus may be worth it is performance. If the performance of my services is significantly better in Incus/LXDs as compared to Docker, then that’s worth it to me. I have not been able to find any type of performance comparison between Incus/LXD and Docker. I don’t know if there are other reasons as to “Incus over Proxmox and Docker” which is why I’m asking the greater community.
Here’s my question:
Based on your experience and taking into consideration my use case (home lab/home use), do the pros and cons of Incus outweigh accomplishing my goal by creating standalone hosts cluster or Proxmox cluster?
I am in no way even slightly an "expert" here, but Incus could be considered a lightweight Proxmox..
They're both going to run some VMs and / or containers, but with Proxmox you get the overhead of the fancy GUI.
So if your host(s) aren't running guests at >90% load, then there won't be any difference in performance.
I've recently installed Proxmox because everyone else uses it (and VMware's free ESXi is now dead)... but after pulling my hair out trying to get some things done, I'm seriously looking to move to Incus.
There's another post here somewhere (the one about free ESXi being killed off) with someone explaining more about Incus, which seems like it's the way to go... maybe worth a search.
I believe you are referencing the same post that got my curious about Incus and started playing around with it.
My biggest gripe is the manual installation of all services which I will do if it’s worth it. So far not sure that it is, hence the post to get more opinions.
There’s is a GUI you can install for Incus but it’s optional and not preinstalled.
I appreciate your input.
You should also consider the time you will spend configuring and setting up everything in Incus.
If you do this with educational purpose go for it, otherwise I will advise you not to, as Proxmox has a wider support and probably finding information, etc. for it is going to be easier.
Alternatively, why don't you dedicate one of the hosts to Incus and play around with it and decide if it works for you or not.
Fair point. I’m most familiar with docker and proxmox. Sorta doing it for educational purposes but I also have critical services (critical to me) running that must be available.
My install is on Arch linux, I just installed incus and cockpit-machines and (from memory) that was enough.
But, yeah, it's definitely a step away from a full GUI...
But I think that's part of the appeal (to me)... there's a lot of things preinstalled with Proxmox, (and XCP-ng, etc), that I'll just never use... so if I could get that old Ras Pi3 as a 2nd node in the cluster without all the fluff... maybe that's a good thing?
That’s another fair point. I do have a couple of pi’s collecting dust. As someone else stated, I need to consider the time it takes me to get up to speed with incus. Can you elaborate on your experience going “from 0 to hero” with incus? Just curious.
I think it is a good way to isolate docker containers from the host without the heavier performance increase of a full VM. Each container can be easily set to an IP address, though the same is probably true for docker idk.
Unrelated, Podman is the a unprivileged implementation of docker with full compatiblity. You can use docker images with it which is great, and the syntax is mostly the same.
Depending on your threat model incus/lxd won't add too much security as they generally use the same background software as docker, leaving things kernel exploits as vulerable as just docker
Haven’t really looked into Podman as I read somewhere (if I remember correctly) that it takes quite a bit of rewrite (from docker compose to podman). Again, might be speaking out of turn here.
I have no problems thus far. It does have a docker compatiblity mode as well