Maybe, one of these days, my SRE team can define SLOs that actually make sense. This "alert on whatever the customer wants" stance really really sucks.
this post was submitted on 01 Apr 2025
28 points (96.7% liked)
Software Reliability Engineering
54 readers
2 users here now
👋 Welcome to the SRE Community!
Whether you're just learning about Site Reliability Engineering/Software Reliability Engineering or you're a seasoned on-call warrior, you're in the right place.
SRE (Site Reliability Engineering - AKA Software Reliability Engineering) is a discipline that uses software engineering principles to ensure that systems are reliable, scalable, and resilient. It’s about balancing feature velocity with system stability—keeping things running even when they shouldn’t.
💬 What can you post here?
Here are a few ideas to get started:
- War stories from production incidents and what you learned
- Cool tools for observability, monitoring, alerting, and automation
- Best practices around on-call, SLOs, blameless postmortems, and chaos engineering
- Questions about reliability engineering and career advice
- Infrastructure as code, CI/CD pipelines, and deployment strategies
- Memes. Tasteful ones. SREs need to laugh too 😅
🌐 Be excellent to each other
This community is part of the programming.dev network. Please make sure to:
- Read and follow the programming.dev code of conduct
- Keep discussions respectful and inclusive
- Assume good faith, and be generous in your interpretations
founded 4 days ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Definitely not that lucky. We have customers who seem to watch dashboards and create a Sev1 anytime latency degrades by 10%. They explain to their account manager that they need to have perfect performance all the time. The AM then comes to us demanding that we increase the sensitivity of the alert. Management agrees. And then, voila, just like that we have an alert that flaps all day and all night that we aren't "allowed" to remove until someone can show that the noise is literally stopping us from catching other stuff.
It's insanity.
EDIT: I only stay because new leadership seems like they want to fix it earnestly. And things are headed in the right direction, but it takes a long time to turn a ship.