this post was submitted on 25 May 2024
10 points (100.0% liked)

Nix / NixOS

1777 readers
1 users here now

Main links

Videos

founded 1 year ago
MODERATORS
 

My solution:

let

  nixFilesInDirectory = directory:
    (
      map (file: "${directory}/${file}")
      (
        builtins.filter
          (
            nodeName:
              (builtins.isList (builtins.match ".+\.nix$" nodeName)) &&
              # checking that it is NOT a directory by seeing
              # if the node name forcefully used as a directory is an invalid path
              (!builtins.pathExists "${directory}/${nodeName}/.")
          )
          (builtins.attrNames (builtins.readDir directory))
      )
    );

  nixFilesInDirectories = directoryList:
    (
      builtins.concatMap
        (directory: nixFilesInDirectory directory)
        (directoryList)
    );
  # ...
in {
  imports = nixFilesInDirectories ([
      "${./programs}"
      "${./programs/terminal-niceties}"
  ]);
  # ...
}

snippet from the full source code: quazar-omega/home-manager-config (L5-L26)

credits:


I'm trying out Nix Home Manager and learning its features little by little.
I've been trying to split my app configurations into their own files now and saw that many do the following:

  1. Make a directory containing all the app specific configurations:
programs/
└── helix.nix
  1. Make a catch-all file default.nix that selectively imports the files inside:
programs/
├── default.nix
└── helix.nix

Content:

{
  imports = [
    ./helix.nix
  ];
}
  1. Import the directory (picking up the default.nix) within the home-manager configuration:
{
  # some stuff...
  imports = [
    ./programs
  ];
 # some other stuff...
}

I'd like to avoid having to write each and every file I'll create into the imports of default.nix, that kinda defeats the point of separating it if I'll have to specify everything anyway, so is there a way to do so? I haven't found different ways to do this in various Nix discussions.


Example I'm looking at: https://github.com/fufexan/dotfiles/blob/main/home/terminal/default.nix

My own repository: https://codeberg.org/quazar-omega/home-manager-config

top 14 comments
sorted by: hot top controversial new old
[–] [email protected] 4 points 6 months ago* (last edited 6 months ago) (1 children)

I do have a function that I cribbed from somewhere, unfortunately I didn't note down the source at the time so can't give them due credit here.

filterNixFiles = k: v: v == "regular" && lib.hasSuffix ".nix" k;

importNixFiles = path: (lib.lists.forEach (lib.mapAttrsToList (name: _: path + ("/" + name)) (lib.filterAttrs filterNixFiles (builtins.readDir path)))) import;

Then used as normal

modules = [ home-manager.nixosModules.home-manager vscode-server.nixosModule ] ++ (importNixFiles ./hosts/required) ++ (importNixFiles ./hosts/machines/${hostname})

Edit

Apologies for formatting on mobile

[–] [email protected] 2 points 6 months ago* (last edited 6 months ago)

Haha, I read this just after trying to figure it out on my own and it seems the solution is kind of similar to what I got, but this looks like it's probably more robust since it extracts it all into a function (though I have yet to understand the syntax 👀). So thanks, I'll be trying that as well!

[–] [email protected] 3 points 5 months ago (1 children)
[–] [email protected] 1 points 5 months ago* (last edited 5 months ago)

I've analyzed the script a bit (..ok for more than 2 hours + 2 of refactoring), because the first time I absolutely didn't understand it, now I've got it, but still, I won't ever understand, why make the syntax so confusing?
My system definition shouldn't be a codegolfing competition (•ˋ _ ˊ•)

TL;DR: I liked your script as a challenge to learn even more and I'm glad I did! Now I know a quite a bit more about the functions that Nix provides and how to use them, not a lot, but better than my previous almost 0, so thank you!

Anyways, here's the unmangled thing explained for anyone else who's interested (wrote for plain evaluation, it's interesting what each part outputs):

{
  /*
  builtins.unsafeGetAttrPos "_" { _ = null; }

  yields:
  {
    column = 46;
    file = "/path/to/this/slightly-unholy-file-finder.nix";
    line = 14;
  };

  you want to get the value of the name (which is the "key" in this key-value list) "file"
  */
  filePath = (builtins.unsafeGetAttrPos "_" { _ = null; }).file; # absolute path to current file
  directoryEntries = builtins.readDir ./.;

  entryNames = map
    (node: "./${node}")
    (
      # get name of files
      builtins.attrNames
        (
          /**
          use the function from before to remove this file right here
          from the set (NOT a list) of nodes found by readDir
          (may be files, dirs, etc.)
          
          Why?
          Because we start reading from the path ./
          which is where this file is located, of course
          */
          builtins.removeAttrs
            (builtins.readDir ./.)
            [
              /*
              get the current file name with some built-in, 
              either un- or poorly documented function black magic fuckery
              (I really wasn't able to find any proper documentation on this function)
              */
              (baseNameOf (builtins.unsafeGetAttrPos "_" { _ = null; }).file)
            ]
        )
    );
}

run it with:

nix eval -f ./slightly-unholy-file-finder.nix

There were multiple problems with this solution as I tried it:

  1. The missing baseName on line 39 which was needed to actually filter out the file path of the script that is being ran, because the paths I got out of readDir were relative (just for me? Did I change something in my environment? I'm not usre, the docs aren't very clear)
  2. It doesn't filter out files that are not .nix files
  3. It doesn't filter out directories (may be intentional though, I personally don't think that's a great idea as far as I got)

I'll post later my own further improved solution starting from my own (tbh, by now more like our) script.

[–] [email protected] 2 points 6 months ago (1 children)

There's the readDir builtin, but I expect nix might complain if you use that

[–] [email protected] 3 points 6 months ago* (last edited 6 months ago)

Oh this is actually great! Searching it up a bit I came across this old Reddit thread that has a solution, so I applied it to my own config which first got me this:

{
  # ...
  imports = builtins.map
    (file: "${./programs}/${file}")
    (builtins.attrNames (builtins.readDir ./programs));
  # ...
}

Then I wanted to refine it to only import .nix files (very much trial and error generated code, I don't even know what I'm doing, but it somehow works lmao):

{
  # ...
  imports = builtins.map
    (file: "${./programs}/${file}")
      (builtins.filter (file: (builtins.isList (builtins.match ".+\.nix$" file)))
      (builtins.attrNames (builtins.readDir ./programs)));
  # ...
}

Thanks for the tip, gotta say I'm very happy! Though I'm wondering how could there not be a simpler solution, it seems like such a common/basic thing to want, maybe what I'm doing is actually undesirable for some reason I'm too noob to understand?

[–] Corbin 2 points 6 months ago (1 children)

At scale, you'll appreciate explicitly spelling out your imports. I currently have 23 importable files, of which two are mutually incompatible (headless vs. Xorg). I don't want a glob over these files because no machine can have all of them; indeed, most machines only have like five imports from the list.

What might be more interesting to you is a common collection of modules which must be imported everywhere. To achieve this, I explicitly declare a commonModules at the top of my flake and reuse it in each machine definition. Another approach might be a common.nix module which recursively contains the common modules as its own imports.

Finally, it doesn't "defeat[] the point of separating" expressions into multiple files to avoid globbing over them. Because NixOS/HM modules are monoidal, they often factor nicely. When you have a dozen different services, you could stuff all of them into one file with one networking.firewall.allowedTCPPorts if you wanted, or you could put each service into its own file and let each module bring its own port to the combined configuration. The latter is easier at scale; I have nine modules declaring TCP ports and five machine-specific TCP ports as well, and it would be a pain to put all of them in one location.

[–] [email protected] 2 points 6 months ago* (last edited 6 months ago) (2 children)

Thanks for the input! I figured there would be a reason why nobody seems to be doing it, but I still struggle to understand, at least for my current use case.
What I'm trying to achieve for now is a solid configuration for my own user on any machine, I'm not trying to (and can't) manage my own system currently as I'm using Fedora Kinoite as the host with only the Nix package manager installed. For now I haven't had the chance to make machine specific configurations but I'm wondering, if on top of how it works now, we could write something like imports = [ ./programs/* ] and have all Nix files in that directory be imported, wouldn't that be a good feature to have? Maybe you do have multiple machines, but maybe you also have several directories from where you will want to import everything regardless of the machine, sure you could make just one file for those if you're not going to make distinctions, but I don't want to put everything in one file because it would just get huge, whereas several files that do one thing are just easier to reason about to me.

common collection of modules which must be imported everywhere

That sounds interesting, do you have any examples I can refer to to know how to do that?

Because NixOS/HM modules are monoidal, they often factor nicely.

What does that mean exactly? I'm not really knowledgeable about functional programming, though that plus the rest of paragraph makes me think of how definitions are "composable" (maybe not the right word) in the sense that you can append and override options that are defined in other imported files without nullifying what was defined in them, is that it?

[–] Corbin 3 points 6 months ago (1 children)

Oh, right, monoids! Yes, you understand correctly.

A monoid is a collection of objects that has some sort of addition and zero. (Depending on your maths background, it might equivalently have some sort of multiplication and unit.) Addition must be associative, and addition with zero must not have any effect. Monoids let us think of a system as built from a sequence of operations; each operation adds to the system, preparing its state incrementally.

Sometimes monoids are commutative, which means that the order of additions is irrelevant to the result. Commutative monoids let us think of a system as built from a collection of operations without worrying about the order in which those operations are applied.

NixOS modules (and HM modules, etc.) are commutative monoids. The zero is {}. The module system lets options declare their own monoids which ride along, like my example of allowedTCPPorts. Because we can combine sets of port numbers (with set union) and get more sets, we can factor a set of ports into many smaller subsets and put each one in their own file. Here's my shortest module, for an internal Docker registry, docker-registry.nix:

{
  networking.firewall.allowedTCPPorts = [ 5000 ];
  services.dockerRegistry = {
    enable = true;
    enableGarbageCollect = true;
  };
}
[–] [email protected] 2 points 6 months ago

I don't have much of a math background, but that makes a lot of sense when talking about the application here! I like the explanation

[–] Corbin 2 points 6 months ago (1 children)

I'm adding some code snippets from my homelab's flake. Minor details are changed. Note how I have a core.nix and also separate files for adding Avahi (zeroconf) and SSH, and for fixing bufferbloat. I could have them as one file, but it's easier to come back to them after several years this way. (bufferbloat.nix was last changed in December 2021, for example.)

I know that some of this code style probably seems weird. Think of it as heavily inspired by Puppet, Chef, Ansible, HCL, etc.; when we are configuring a system, it is very very nice to be able to comment out a single line at a time.

Click to see code!Some common modules, bundled into a NixOS module:

    commonModules = {
      imports = [
        nixpkgs.nixosModules.notDetected
        ./avahi.nix
        ./bufferbloat.nix
        ./core.nix
        ./ssh.nix
      ];
      nix.registry.nixpkgs.flake = self.inputs.nixpkgs;
      nixpkgs.config.packageOverrides = pkgs: {
        mumble = pkgs.mumble.override {
          pulseSupport = true;
        };
      };
      users.users.corbin = {
        isNormalUser = true;
        extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
      };
    };

A NixOS machine definition:

      joker = nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [
          commonModules
          ./joker/configuration.nix
          ./gl.nix
          ./sound.nix
          ./wifi.nix
          ./xserver.nix
        ];
      };

[–] [email protected] 2 points 6 months ago (1 children)

I see, that's really neat!
So joker is the name for one of the machines right? If so, how do you select that particular section in the actual machine's config?

Also, the code style seems normal to me? I'm not very familiar with Nix though, so maybe that's why I don't spot the weirdnesses

[–] Corbin 2 points 6 months ago

The flake exports look like outputs.nixosConfigurations.joker, each one matching a hostname. There's a poorly-documented feature of nixos-rebuild where you can point it at a flake with --flake and it will try to use the configuration matching the current hostname. So, I make the flake available on each machine and customize it using the hostname. One flake, a dozen machines. Works well enough for a homelab but would not work for a large cloud deployment.

[–] [email protected] 1 points 5 months ago

Thanks again, everyone!!!

btw, @[email protected], just updated my post with my solution as promised :)