this post was submitted on 13 Apr 2024
8 points (90.0% liked)

DevOps

1710 readers
1 users here now

DevOps integrates and automates the work of software development (Dev) and IT operations (Ops) as a means for improving and shortening the systems development life cycle.

Rules:

Icon base by Lorc under CC BY 3.0 with modifications to add a gradient

founded 2 years ago
MODERATORS
Ategon
RandomDevOpsDude
8
Self Hosted SCM & CI/CD Chicken and Egg (self.devops)
submitted 8 months ago by Senal to c/devops
11 comments fedilink hide all child comments
 

Struggling with a problem that i just can't seem to figure out.

When starting from scratch self hosting both the SCM and CI/CD server.

Given that you can't use an existing setup to deploy/manage it, what is the best practice for deploying said services?

top 11 comments
sorted by: hot top controversial new old
[–] alienscience 5 points 8 months ago (1 children)

I am not sure if it is best practice, but this is what I do and it might provide some inspiration:

  • Bootstrap from a private gitlab.com repository with a base ansible setup. Executed from a laptop.
  • The bootstrap setups up k8s and installs a bare bones git repository docker container based on https://codeberg.org/al13nsc13nc3/gitsrv.
  • Flux CD is installed into the bare bones git repository and k8s.
  • Flux CD is used to install Forgejo and Woodpecker CI using the bare bones git repository as the gitops source of truth.

This has the advantage that Gitops and normal git repositories are separate. I think that a similar principle would work with docker compose instead of k8s.

[–] Senal 1 points 8 months ago

I have also done this previously but i'll not be using third party hosted SCM or CI for this one so it kinda rules this approach out. (I'll edit the main post to reflect this)

Looks like manual/locally-scripted is the way, just wondered if there was something more.

[–] [email protected] 4 points 8 months ago (1 children)

Isn't this what bootstrapping is for? Manually set up the system to the point it can be taken over by ci/cd

[–] Senal 2 points 8 months ago

Isn’t this what bootstrapping is for? Manually set up the system to the point it can be taken over by ci/cd

Indeed, was just wondering if there was some industry standard i was missing that was a bit more managed.

No worries if not.

[–] icb4dc0de 2 points 7 months ago (1 children)

I tend to do manual bootstrapping with local execution as well but when using Forgejo/Gitea and corresponding actions you could also init the repo, start building your pipeline and use act to run it (locally at first if course) and as soon as you have your infrastructure in place you could continue to use the same pipeline there?

Still not ideal/perfect but if you don’t want to depend on some SaaS then this at least already runs the automation as it will be later?

[–] Senal 1 points 7 months ago (1 children)

Yeah, I'm going with a tiny dedicated infra bootstrapping box with all the tools I'd need to bootstrap the main infrastructure.

Using a hypervisor (proxmox in this case) I have some prebuilt vms's and container images that I can use for the bootstrap instances so i'd not need to completely hand roll it again should it be needed.

I'm looking at cloudinit scripts to see if that's useful for this.

I really like packer but I'm hesitant to rely on anything hashicorp until whatever they have going on shakes out.

Then I just load up the bootstrap box with the main infra code and use woodpecker to deploy.

Code and config backed up, also mirrored to newly created infra forgejo instances, just in case.

If I can get a semi presentable cloud init based bootstrap system working nicely I'll stick it somewhere people can get to it, in case it's useful to someone else.

[–] icb4dc0de 1 points 7 months ago

If you don’t mind I’d be interested in how the cloudinit for proxmox looks like 😁 So far I only used it for Azure and Hetzner Cloud to bootstrap Coder machines but from your description I’d expect that should work. Very cool approach!

[–] RonSijm 2 points 8 months ago (1 children)

Struggling with a problem that i just can’t seem to figure out.

What problems are you struggling with specifically?

You basically just pick a system, for example Forgejo - that's comparable to a self-hosted github. Which also comes with github-like actions for CI/CD/Building

If those actions are not good enough for building, you could also self host something like Jenkins or TeamCity

[–] Senal 1 points 8 months ago (1 children)

What problems are you struggling with specifically?

You basically just pick a system, for example Forgejo - that’s comparable to a self-hosted github. Which also comes with github-like actions for CI/CD/Building

I can deploy these by hand sure, but is that the only way ?

Let's assume forgejo and woodpecker.

I'd need to spin up each service + the db (postgres probably) for each.

Given i'd not have an SCM system or build pipelines until after they were deployed, am i just doing it by hand and hoping for the best or working with something like ansible, saving the scripts to a folder somewhere and manually running them myself?

How about future maintenance or reproducibility?

I'm fully capable of doing it by hand and not against it, just wasn't sure if there was a commonly used bootstrapping mechanism i wasn't aware of.

[–] [email protected] 2 points 8 months ago

Ansible files stored locally to bootstrap, then the same set of ansible files gets put in a repo once the repo is up.

IMO

[–] melezhik 1 points 6 months ago* (last edited 6 months ago)

my 2 cents here, though I don't understand all the context, you might take a look at sparky - which is lightweight task runner with web console, so you may throw a bunch of jobs into it to do all the "bootstrapping" so that you may later repeat the same if required on any fresh environment ...