this post was submitted on 07 Jun 2024
22 points (89.3% liked)

Python

6466 readers
21 users here now

Welcome to the Python community on the programming.dev Lemmy instance!

πŸ“… Events

PastNovember 2023

October 2023

July 2023

August 2023

September 2023

🐍 Python project:
πŸ’“ Python Community:
✨ Python Ecosystem:
🌌 Fediverse
Communities
Projects
Feeds

founded 2 years ago
MODERATORS
 

I happen to write a lot of python code dealing with git repositories. Currently I am calling the git command line tool from python and interpret the output.

This solution really doesn't scale well. Can you recommend a python library that wraps git functionality?

I have found three:

  • GitPython: Seems to work well, but it is in maintenance mode, unlikely to be improved. It also does not have any type hints making working with it annoying.
  • pygit2: Seems well supported and has type hints. But it also seems very low level and pretty tedious to use.
  • dulwich: Looks very promising feature wise but I'm unsure how well it is supported. It seems like an ambitious project being largely done by just one person.
top 13 comments
sorted by: hot top controversial new old
[–] [email protected] 4 points 6 months ago (1 children)

I read through the docs of pygit2, how is it too low level compared to using direct console output?

if you need complex workflows, couldn't they be built over the convenience of the library?

[–] [email protected] 5 points 6 months ago

Maybe pygit2 is indeed the way to go. When I looked into it a while back it looked very low level like it only implemented the git plumbing. But maybe I looked at the wrong part of the docs, because it doesn't look too bad.

[–] [email protected] 2 points 6 months ago

Dulwich is decent. Has some good porcelain functions. But it's organized kind of weird. I sort of recall it's the only one that isn't a wrapper on the git CLI?

Anyway, they all kind of suck in my experience.

[–] lordmauve 2 points 6 months ago

I recommend wrapping the git cli commands using subprocess, using porcelain output modes etc, and parsing the output.

We have had stability problems with GitPython (which wraps gitdb). On Linux gitdb does clever things with sliding mmap, which caused some crashes (in a multi threaded environment), and I found simple race conditions in the code for writing loose objects, which is about as simple an operation as can be, so I lost faith with it. I do use gitdb in one read-only single-threaded system; it's undoubtedly fast.

The biggest issues with git libraries are around the complexity of git configurations. Any independent reimplementation is probably going to support the most common 99% of features but that 1% always comes back to bite you! We use a lot of git features in service of a gigantic monorepo, like alternates and partial clones and config tricks.

If we use command-line git we get 100% compatibility with all git configuration and ODB features, and it's hard to ensure that with an independent git implementation (even libgit2).

When you say "that solution doesn't scale well" - we have made it scale. git itself scales well for operations it can perform natively, you just have to use the features effectively, often the high-level operations but sometimes lower-level commands like git cat-file --batch, git mktree --batch, etc. It's not as fast as gitdb but fast enough, and I can have high confidence that I can write something once and it won't break or cause problems later.

[–] [email protected] 1 points 6 months ago (2 children)

Honestly if you already know all the git commands? I’d use sh

[–] [email protected] 6 points 6 months ago (1 children)

You'd still only get strings as returns. No objects modeling git concepts.

[–] [email protected] 1 points 6 months ago* (last edited 6 months ago)

Fair point, I usually use exit codes

[–] [email protected] 1 points 6 months ago

We have historically used GitPython a lot, but in a recent project I tried git via sh instead. It works great. If you already know the git cli, this feels very ergonomic to use.

[–] [email protected] 1 points 6 months ago
[–] Corbin 1 points 6 months ago

With no more details? I'd go with Dulwich. libgit2 is overly picky about inputs and can't be hacked apart at all, and this affects its bindings too. I recently found myself monkey-patching Dulwich to allow otherwise-forbidden characters in refs, and this would have been fundamentally impossible with anything on top of libgit2.

[–] [email protected] 0 points 6 months ago

I usually use subprocess. Python has a very nice API for calling subprocesses.

[–] [email protected] 0 points 6 months ago

I used pygit2 a few years ago and it was easy. Can’t complain.

[–] [email protected] 0 points 6 months ago

Sounds like pygit2 is the move until dulwich has a bit more support.