I want to reset my server soon and I'm toying with the idea of using a different operating system. I am currently using Ubuntu Server LTS. However, I have been toying with the idea of using Fedora Server (I use Fedora on my laptop and made good experiences with it) or even Fedora CoreOS. I also recently installed NixOS on my desktop computer and find the declarativeness pretty cool (but I'm still a complete beginner) and could imagine that it would fit well into a server setup.
I have quite a few services running on my server, such as Nextcloud, Conduit (Matrix), Jellyfin, etc. and all in containers. I would also rather not install programs without containers, because 1. compose is super easy to maintain and set up, 2. it remains very clear with containers (and compose) and 3. I believe that containers are more secure. But since I also want to make the services inside the containers available, I currently have Nginx installed as a reverse proxy (not in the container, but on the system) and always create certificates with certbot so that I can use HTTPS encryption.
In the paragraph above I actually described exactly the use-case of Fedora CoreOS, but I have no experience with the system and how it works. That's why I'm still a bit hesitant at considering the OS at the moment.
I can imagine that NixOS with its declarative nature seems well suited, since, as I have heard, you can configure containers as well as Nginx and with Nginx also https certificates declaratively.
But I could also use a base system like before (Fedora Server or Ubuntu Server) and simply install podman, nginx and certbot and manage everything that way.
Have you had any experience with Fedora Server, Fedora CoreOS, NixOS or a completely different operating system for servers and what are/were your impressions with this setup? Or do you just want to share your knowledge here? I would be delighted.
Firstly, I don't think Docker is intended to be used as a security layer. I could be wrong, but I think it's relatively easy to escape the sandbox. Although that might be different nowadays.
Anyway, I recently switched from Debian to NixOS for my server and it's so much nicer. Being able to use a single language for configuring everything all in one place is so nice. If I want to try something new, I can just throw maybe 30 lines in a config and it's there. If I don't like it, then I can remove that config and it's gone. Most services you'd want to run on a server are available in the package manager, and many have rather sensible defaults.
The only headache for me was the lack of documentation (especially with flakes). There's also a learning curve with the Nix language itself.
I think it does serve as a security layer. Like if you have vulnerable software running in a docker container, and people use that to gain "root access" - initially they're just inside the docker. Of course you need us to proper docker setup, like not just mount your entire file system into your dockers "because it's easier" and that kinda stuff.
There are a bunch of ways for Docker Breakout / Privilege Escalation - but still, it adds an extra step instead of having access to the entire system immediately. And again, it's just a matter of hardening your docker, and closing these open holes.