this post was submitted on 11 Dec 2023
44 points (95.8% liked)

Programming

17526 readers
286 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]

founded 2 years ago
MODERATORS
snowe
Ategon
[email protected]
44
Tool to manage CLI tools (lemmy.world)
submitted 11 months ago by [email protected] to c/programming
37 comments fedilink hide all child comments
 

I'm trying to find a thing, and I'm not turning up anything in my web searches so I figure I'd ask the cool people for help.

I've got several projects, tracked in Git, that rely on having a set of command line tools installed to work on locally - as an example, one requires Helm, Helmfile, sops, several Helm plugins, Pluto, Kubeval and the Kubernetes CLI. Because I don't hate future me, I want to ensure that I'm installing specific versions of these tools rather than just grabbing whatever happens to be the latest version. I also want to ensure that my CI runner grabs the same versions, so I can be reasonably sure that what I've tried locally will actually work when I go to deploy it.

My current solution to this is a big ol' Bash script, which works, but is kind of a pain to maintain. What I'm trying to find is a tool where I:

  • Can write a definition, ideally somewhere shared between projects, of what it means to "install tool X"
  • Include a file in my project that lists the tools and versions I want
  • Run the tool on my machine and let it go grab the platform- and architecture- specific binaries from wherever, and install them somewhere that I can add to my $PATH for this specific project
  • Run the tool in CI and do the same - if it can cache stuff then awesome

Linux support is a must, other platforms would be nice as well.

Basically I'm looking for Pythons' pip + virtualenv workflow, but for prebuilt tools like helm, terraform, sops, etc. Anyone know of anything? I've looked at homebrew (seems to want to install system-wide), and VSCode dev containers (doesn't solve the CI need, and I'd still need to solve installing the tools myself)

top 37 comments
sorted by: hot top controversial new old
[–] [email protected] 30 points 11 months ago (4 children)

Is this Nix bait? Cause I'd say Nix.

[–] [email protected] 6 points 11 months ago

It wasn't intended as bait, and TBH I'd forgotten that Nix existed outside of NixOS. Something for me to look into I guess

[–] [email protected] 6 points 11 months ago

I’d say Nix as well!

[–] [email protected] 5 points 11 months ago (4 children)

Very briefly playing about with Nix it does seem pretty compelling - only issue I can see is I don't seem to have a straight forward way of installing a specific version of a tool from the official repo - you get whatever the current version is that the package maintainers have published for the specific snapshot you are using. I guess I could maintain my own packaging for different versions if it turns out to be important

[–] [email protected] 5 points 11 months ago

You can hardcode a specific version of nixpkgs, instead of a branch. With the new Nix CLI & flakes enabled you can do something like this:

nix run "github:NixOS/nixpkgs/b4372c4924d9182034066c823df76d6eaf1f4ec4#cowsay" "moo mooooooo"

That's the commit I'm seeing for nixos-23.11 today, and it should still give you that exact version of cowsay years from now.

Of course, the better option is to make a dev shell with flakes. Flakes come with a lockfile builtin that accomplishes the same effect, and there's no problems having different projects on different lockfiles/versions. It's a bit more work to learn, the Zero to Nix tutorials are pretty decent at teaching and come with examples though (ultimately most things are ~30 lines of boilerplate and a list of packages that you want).

[–] [email protected] 3 points 11 months ago (1 children)

I only started diving into nix this year so I'm still learning, but yeah, I'm pretty sure the lack of granular versioning is a common pain point with nixpkgs. I'd suggest checking out flakes if you haven't already, but be warned, it gets hairy lol

[–] pkill 3 points 11 months ago (2 children)

It is not? At one of my previous jobs it was nix that allowed me to get a compatible legacy version of kubectl up and running easily iirc

[–] [email protected] 3 points 11 months ago* (last edited 11 months ago) (1 children)

It definitely still is. I use nix on the daily and specialize in old versions seach; Just because it's very possible doesn't mean people don't find it to be a pain point.

With the right tools (which are not easily found in the docs or on Google) finding one old version is fine. Running one old version is flawless. But mixing that old version with newer versions of other packages causes problems because of the nix LD_LIBRARY_PATH issue.

[–] pkill 1 points 11 months ago (1 children)

Eh so probably got lucky with the fact that the software I needed was statically linked

[–] [email protected] 2 points 11 months ago* (last edited 11 months ago)

I was lucky for a while too, since you can also get lucky with dynamically linked libraries. Sometimes they find the new version of the .so (from other packages) and it works, but sometimes it finds a system .so and works until there is a system update. Which ruins the whole reproducability thing, although using the sandbox options of nix can help with this.

Nixpkgs is better about patching the RPATH now, but that's the thing; using old versions is like going back in time. We'd need to go back in the git history and also patch the super old version.

There are tools like nix-ld which can help, but they need to be setup and they've got edgecases too.

[–] [email protected] 1 points 11 months ago* (last edited 11 months ago) (1 children)

For some reason I thought it was more annoying to work out than it looks to be. @[email protected] you might want to check out nix-versions

[–] [email protected] 3 points 11 months ago* (last edited 11 months ago)

Actually Lazamar's is kind of out of date. Here's the best sites for it:

And I made an interactive Cli tool that let's you search all four of those simultaneously!

[–] [email protected] 3 points 11 months ago* (last edited 11 months ago)

Use this! nvs --install ruby@2 (I'd attach the gif but lemm.ee doesn't support images)

I had your exact complaint. And it only took me 3 years and hundreds of hours of learning nix to make that tool 😅

Btw, while it solves version search, fair warning you're going to immediately run into other usability problems. I use nix every day but I don't gaslight people into thinking it's usable.

[–] ericjmorey 1 points 11 months ago

If you're using the Debian "no Frankenstein" approach, you're limited to what is packaged, but if you do your own thing, you can have it build/install whatever you want.

[–] metacat 2 points 11 months ago

Nix is interesting and fine if you are a solo developer with time to learn the details. I did try it briefly but find the learning curve of rtx is near zero - it does less than Nix but is very easy to adopt in a team. Compatible with asdf but faster and easier in some way, and actively developing.

[–] [email protected] 13 points 11 months ago (2 children)

I think what you are looking for can be found in rtx or asdf, I have used both for what you are describing, even those same tools hah. I’m currently using rtx more than asdf though for newer projects. I have been tempted by nix though, might take that plunge soon.

[–] metacat 5 points 11 months ago (1 children)

I recently started using rtx and I’m a big fan.

Works for a long list of tools and makes it very easy to remove lots of install commands and just do “rtx install” in any directory to install and switch to right versions of a set of tools. Also works great on CI using same configuration files.

[–] [email protected] 2 points 11 months ago* (last edited 11 months ago)

Rtx on/off meme template;

"Off" is a greyscale picture of metacat struggling with manual dependencies. Possibly combined with "Isn't there a better way?" from ye olde shopping channel ads.

"On" Is a HDR image of a smiling metacat having no such issues. In the background a rainbow can be seen over a flowery garden, in which people are holding hands with rabbits frolicking in the grass.

Yeah yeah, I know, I'll go to bed now...

[–] [email protected] 2 points 11 months ago (1 children)

Ended up going with rtx - thanks for the suggestion!

[–] [email protected] 2 points 11 months ago

Awesome! You bet, glad to hear it worked out.

[–] bloopernova 12 points 11 months ago* (last edited 11 months ago) (1 children)

Look into asdf: https://asdf-vm.com/ or rtx: https://crates.io/crates/rtx-cli

One .tool-versions file in a directory and you get specific versions of each tool you want.

EDIT: Oh and both integrate with direnv very well, which can also do per-directory env vars and scripts: https://github.com/direnv/direnv

[–] [email protected] 3 points 11 months ago

definitely this!

[–] snowe 9 points 11 months ago

Asdf is what you’re looking for. We use it in every repo and it manages every tool version with no issues.

[–] [email protected] 8 points 11 months ago (2 children)

I have no knowledge in this area, but this really sounds like containers territory (Docker, not VSCode) ?

[–] [email protected] 3 points 11 months ago (1 children)

Dev containers is a Docker container under the hood, it's just some tooling to make it slightly easier to set up - same objections, if I'm using Docker, I'm still stuck with a gross script, just that it's running inside the container rather than directly on my machine

[–] Wyatt 6 points 11 months ago* (last edited 11 months ago) (2 children)

What do you mean? I containerize my tools, and write a docker file to install everything. The dockerfile is just something like:

FROM ubuntu:22.04

RUN apt-get update &&
apt-get install -y helm pluto kubeval etc

How is that gross?

[–] [email protected] 2 points 11 months ago

Now install tools that are only available as github released binaries. And ensure that hashes match for that. Maybe install a tool that needs to be compiled.

[–] [email protected] 1 points 4 months ago

Just had the same setup and that project was randomly breaking as some dependencies were not quite stable upstream. If you do this, please pin the versions like so: sudo apt-get install apache2=2.2.20-1ubuntu1 or you'll have fun debugging in the future ;) But I think something like nix or devbox would be better overall.

[–] [email protected] 1 points 11 months ago* (last edited 11 months ago)

It's news to me that we're not all just abusing Docker for this. I'll have to look into some of these mentioned tools.

[–] [email protected] 5 points 11 months ago* (last edited 11 months ago) (1 children)
  • Pkgx (formerly Tea) from the creator of homebrew
    • it's adsf but professional instead of crappy (yes, shots fired, I've used asdf for a long time)
  • devbox for better reproduciblity than Pkgx
  • And yes Nix (obligatory; BTW I use nix btw btw). Nix is supposed to be exactly what you're asking for and has unbeatable reproducability. But it's simply not ergonomic enough yet. I've been deep diving into it for 3 years and it's still painful to setup a project with it. Devbox uses nix under the hood but kinda of abuses nix. So it looses some of the guarantees but gains being usable/ergonomic today.

Full disclosure, I use nix (not devbox) for all my stuff cause I care about hardcore reproduciblity.

[–] [email protected] 1 points 11 months ago (2 children)

What's your issue with asdf? It works really well for me

[–] Andy 1 points 11 months ago

I'm not who you asked, and not a user of pkgx, but one of the reasons I prefer rtx (which supports asdf plugins) to asdf is that by default it doesn't use shims, but updates the PATH instead.

So for example, running which python will show me a clearly versioned python executable path, rather than a mysterious shim that represents a different realpath at different times.

[–] [email protected] 1 points 11 months ago* (last edited 11 months ago)

It has random side effects that break cli tools (tools that are not even installed with ASDF)

The more versions you add, the slower it gets. And it can get really slow (that issue was opened in 2018, and it is fixable).

It's got many "well it works on my machine" problems. And the author said that's a "wontfix" design choice. That's fine for the author, it's FOSS. But it means my workflow is going to randomly break and I just can't have that when it's my job.

Pkgx, Devbox and nix avoid all these things.

[–] [email protected] 2 points 11 months ago

https://devenv.sh/

Exactly what you're looking for.

[–] [email protected] 2 points 11 months ago (1 children)

Thanks for the suggestions. As an update:

  • I spent a few hours playing about with Nix; it's really cool, definitely a very interesting idea but the "time required to learn the tool" to "time saved by using the tool" to "time spent fixing things when it turns out you don't know the tool as well as you thought" ratio isn't looking great right now
  • I've reworked one of my repos to use rtx - it's not perfect, but it's doing what I want without much fuss, so probably going to go with this for now
[–] Andy 1 points 11 months ago (1 children)

Great! If you get a chance, I'd be interested to hear about your rtx complaints.

[–] [email protected] 2 points 11 months ago

Tbh, I think my issues are less with rtx itself and more the plugins I'm using and the constraints imposed by the asdf plugin structure.

I ran into issues where poorly written plugins could fail to install, but rtx wouldn't recognise the issue and if I re-ran "rtx install" a second time it would tell me that the runtimes were all up to date. I'll see if I can put together a GitHub issue describing it in more detail.

It'd be nice if there was a simple way to reference a local directory as a plugin for a project rather than having to publish a separate git repo.

Generally it's a really well built tool and the docs are excellent, my complaints are nitpicks