My EEEEEEEEEEYES AUUUUUUUUUUUUUUUUUUGH the instance is pure white, probably utilizing some witch-craft to bring sunlight directly into my eyes. How do I return the blessed darkness to my screen??
this post was submitted on 10 Jul 2023
33 points (94.6% liked)
Lemmy.ca's Main Community
2814 readers
1 users here now
Welcome to lemmy.ca's c/main!
Since everyone on lemmy.ca gets subscribed here, this is the place to chat about the goings on at lemmy.ca, support-type items, suggestions, etc.
Announcements can be found at https://lemmy.ca/c/meta
For support related to this instance, use https://lemmy.ca/c/lemmy_ca_support
founded 3 years ago
MODERATORS
Sounds like beehaw.org has shutdown temporarily just to be safe, sounding like a vulnerability in Lemmy.
Currently it seems to be a vulnerability with custom emojis only, which this instance never had, so currently we shouldn't be affected. However this is a developing situation and we will continue to monitor.
I just want to say I love you guys
Thanks!
Good job admins for staying on top of the situation!
(... and have you made a snapshot backup, just in case? 😁 )
I can't remember exactly but @[email protected] is the one that takes care of those things, I think he's got it set up to backup every 8 hours? I'm taking a rapid crash course in sysadmin but I'm not really ready to start poking at production server stuff yet.
Seems they're back up and the devs pushed out a patch for the vulnerability.
We have updated to a patched version.
It's been interesting watching this all play out on an open source social network. It's all out in the open so it caused quite the drama, but the actual order of events? Site goes down and is back up and vulnerability patched like 4 hours later? That's really impressive.
Power of the open source community.
In my opinion the "drama" was a critical part of immediately drawing attention to the voulnerabilty and bringing it to the attention of most instance admins very quickly.
Few things that have been added on my to-do list that I've learned from this.
- We need more backend man power for coverage.
- Major instances, and probably all instances should partner with another instance that's in an opposite time zone for emergency response. Ideally having partnered admins and backend admins with no more than 8 hours difference between each one for 24 hour reliable coverage would be ideal. Partnered admins should in my opinion have each other's phone numbers and have it set to bypass do not disturb.
- We need to make sure users know how to contact admins off Lemmy for emergencies, as well as ensure that admins are tagged when a situation like this develops. (To my knowledge no lemmy.ca admin was tagged when this started to unfold.)
- There's more thoughts but I can't remember them on 5 hours of sleep 😴
Any additional suggestions are welcome!
Is there a lemmy.ca mastodon account or something as an alternate place to contact/get updates?
No, but that's a great idea. Thank you.
What the hell happened here?? I get logged out of wefwef, come here to investigate, and I see something about a vulnerability???
view more: next ›