this post was submitted on 11 Aug 2023
22 points (100.0% liked)

Git

2910 readers
1 users here now

Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

Resources

Rules

  1. Follow programming.dev rules
  2. Be excellent to each other, no hostility towards users for any reason
  3. No spam of tools/companies/advertisements. It’s OK to post your own stuff part of the time, but the primary use of the community should not be self-promotion.

Git Logo by Jason Long is licensed under the Creative Commons Attribution 3.0 Unported License.

founded 1 year ago
MODERATORS
Ategon
canpolat
jnovinger
22
Git scraping: track changes over time by scraping to a Git repository (simonwillison.net)
submitted 1 year ago by canpolat to c/git
5 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (1 children)

And that's just a single repository. Although for a single JSON scrape it shouldn't take a full minute

[–] [email protected] 2 points 1 year ago

According to the Actions log for that repository it generally takes about 11-16 seconds per scrape, nowhere near a full minute. Worst case, if it's 16 seconds every time, for a 31-day month that's 31 days * 72 actions/day * 16 sec/action = 595.2 minutes, well within the free tier limit.

And if it does become a problem it can easily be scaled down to once per hour or something with a single git commit.