this post was submitted on 16 Sep 2024
34 points (100.0% liked)

FreeAssembly

75 readers
6 users here now

this is FreeAssembly, a non-toxic design, programming, and art collective. post your share-alike (CC SA, GPL, BSD, or similar) projects here! collaboration is welcome, and mutual education is too.

in brief, this community is the awful.systems answer to Hacker News. read this article for a solid summary of why having a less toxic collaborative community is important from a technical standpoint in addition to a social one.

some posting guidelines apply in addition to the typical awful.systems stuff:

(logo credit, with modifications by @[email protected])

founded 7 months ago
MODERATORS
 

I'm in our daily standup and it's turned into exchanging fucked up sysadmin redundancy tales.

One place I worked lost a machine room. They'd fired people so fast that nobody remembered where the boxes were any more.

I knew, but they didn't ask me. Oh well!

The cycle of IT binge and purge is eternal. Post your tales here.

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 16 points 2 months ago (1 children)

Company Y: The main datacenter was on tower 2 and the backup datacenter was on tower 1. Most IT staff was present when the planes hit.

Well, that one dark "redundancy" story...

[–] [email protected] 7 points 2 months ago (5 children)

I don't understand why they had redundancy so physically close.

Whatever affects one has a high risk of affecting the other.

Different regions is a thing for a reason.

[–] [email protected] 12 points 2 months ago* (last edited 2 months ago)

I think the OP was talking about the other "redundancy", as in "your whole team has been made redundant". In this context your story is very dark indeed :D

[–] [email protected] 7 points 2 months ago

My old company has both data centers in the same metro area. A lot of handwaving is involved when you ask about events that can take out a metro area (natural or man made).

[–] [email protected] 6 points 2 months ago

"~~perverse incentives~~ budgets rule everything around me"

[–] [email protected] 5 points 2 months ago

It's probably good to situate in time when thinking about these things. The twin towers were how a lot of companies became examples of what location redundancy really means. These days people are keeping that lesson well in mind, but back then, not so much.

[–] [email protected] 2 points 2 months ago

There are tradeoffs to higher and higher grades of redundancy and the appropriate level depends on the situation. Across VMs you just need to know how to set up HA for the system. Across physical hosts requires procuring a second server and more precious Us on a rack. Across racks/aisles might sometimes require renting a whole second rack. Across fire door separated rooms requires a DC with such a feature. Across DCs might require more advanced networking, SDN fabrics, VPNs, BGP and the like. Across sites in different regions you might have latency issues, you might have to hire people in multiple locations or deal with multiple colo providers or ISPs, maybe even set up entire fiber lines. Across states or countries you might have to deal with regulatory compliance in multiple jurisdictions. Especially in 2001 none of this was as easy as selecting a different Availability Zone from a dropdown.

Running a business always involves accepting some level of risk. It seem reasonable for some companies to decide that if someone does a 9/11 to them, they have bigger problems than IT redundancy.