Lemmy.ml did a server migration which was posted about 8 hours ago which was pinned on lemmy.ml meta. It appears in the comments that some users have experienced a 502 error, which was related to “iCloud’s Private Relay”. They were able to bypass the issue using a VPN, although I haven't had any issues accessing lemmy.ml on my pc. I also read something about AWS us-east-1 being down, so it could also be related to that.
this post was submitted on 13 Jun 2023
26 points (100.0% liked)
Technology
37712 readers
164 users here now
A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.
Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 2 years ago
MODERATORS
us-east-1 must not be completely down, or I'd be having a much worse day!
Saw it on https://lemmy.world/post/105778
Ah, their incident page says it was mostly lambda, my infra is mostly all on EKS/EC2.
Ahh, that makes sense, well lucky you then haha
yeah our instance is having problems too with lemmy.ml communities
Following communities could be another issue entirely, sometimes you have to subscribe then unsubscribe multiple times in order to get it to work.
@Sleeping I can't subscribe in the 1st place. Guess I'll just have to note down the community and try subscribing tomorrow again as it's late around here 😁
Oh! Hmm, I've never run into that issue before. Although I have noticed on some instances it works faster and more reliable than others.
@Sleeping I am on Friendica. It shows like this.
I've never used Friendica, so I'm honestly not sure. Maybe this could be related to the ipv6 issue that the instance hoster would need to fix.
It's weird being part of broad communities like technology or games again. On reddit I unsubbed all the default subs and subscribed to more niche things like 3D printing or weird games. It's quite a change to see people talking about things like mainstream AAA games. Intellectually I knew that a lot of people had to get hyped about the latest Bethesda game or Apple products or whatever, but I think part of me stopped believing it, because it still surprises me whenever I see it.
This appears to have somehow ended up in the wrong spot. I hadn't even clicked on this thread, and yet here's my reply to a different one. Ah well.
I've seen a couple of people mention this bug. I'll check if there's an issue in their bugtracker for it.
Retro games and patient games were pretty good subs.
Yeah, I'm a mod of retrogaming and a few few other subs in that network. Best group of users ever. 320k people in just the one sub and I think as a team we had to take down maybe 3-4 posts a day? Maybe? I banned 2-3 people in a month, that's absolutely unheard of in a sub that size. I think the worst argument I saw the whole time was a running "the playstation one was not called the PSX" "Yes it was" "NO IT WASN'T!" "HERE IS AN ARTICLE, YES IT WAS." thing that we had to specifically make a rule about. Odd little drama, but other than that? No fights at ALL. I'm talking a report maybe every few days for something being off topic if we didn't get to it quick enough.
The niche subs about fringe interests, even if they're popular ones, often bring out the very best people. And the patient/retro gamers are so very chill.
Not sure where they host but AWS is currently in the middle of a pretty big outage so it could be related.
After doing a WHOIS, it appears that Lemmy.ml is hosted on OVHCloud.
Edit:
As follow up on that, OVH cloud does appear to be doing some maintenance on their infrastructure: https://www.status-ovhcloud.com/
It looks like the fix for this is to disable IPv6 - not all of us can do that (or want to do that).
Lemmy.ml will need to fix their AAAA record likely.
In short, yes it will take longer to "catch up".
The longer answer, be patient. This is an issue that will be resolved. I'm sure that the admins are doing their best to get the instance up & running ASAP.
From my experience during the great Twitter migration to Mastodon, you saw the same thing. Especially the "main instance(s)", mastodon.online and mastodon.social in this case, were overloaded to the brim. After a day or two, the sysadmins and developers were able to stabilize the instance.
The sysadmins in this case were responsible to keep the servers up and running (and scale up where needed). The developers because their code hasn't been "battle tested" for this level of interaction. So changes were needed, to optimize and squash bugs.
I hope this helps 😊
Edit: fat fingers, ie fixed typo
This is a fantastic point. I watched several migration waves from Twitter to the Mastodon/*oma/*key network over the course of Nov-Feb and each became less impactful as instances and developers understood better how that traffic would flow. Consider also that those platforms had also been around for a few years already.
Lemmy and kbin are very, very young still, so it will be great to see them develop over the course of the coming months. I expect the next couple weeks leading to the June 30 3PA closures on Reddit will be spent preparing for another wave.
Oh definitely that the next couple of weeks will be spent preparing for a new wave. I wouldn't be surprised if that was the reason why Lemmy.ml was down. Yes I know that it was because of a server migration but, that server migration also didn't happen for no reason ;-)
The first migration wave to Mastodon was very bad. I still remember, that following a person was nearly impossible. Not to mention the fact that, even if you could follow someone, it didn't automatically meant their messages ended up in your feed.
I know for example that version 0.17.4 of Lemmy (that was released 2 days ago) already has some database optimizations.
Used Reddit for 13 years, tried out Kbin and Lemmy yesterday and settled on Lemmy.
Long story short, I'm going back to Reddit.
- There needs to be ONE site, Lemmy.com, that people goto. This entire thing about having .whateveryouwant is VERY off putting. Most internet users have been trained to be extremely wary of odd or unusual things, so having anything besides .com/.net/.org will turn away a huge portion of users.
I initially setup an account on Lemmy.world, then realized that I couldn't migrate it to another server and that when I deleted that account on that server all my comments were deleted.
-
Deciphering the distributed nature of it took me, a relatively tech-friendly person, almost the entire day and several 'What the fuck?' posts. I now understand it more. There are some very low-level guides that have been haphazardly put together, but there absolutely needs to be a MUCH smoother guide/explanation to this whole thing. That learning process will turn people away for sure.
-
BECAUSE I understand it more now, I'm left feeling VERY uncomfortable about my data security. If this is going to become a mainstream thing, as it reaches and before it gets to that critical mass of users, there's going to be SO. MANY. SECURITY ISSUES. There's no 2fa at all, hacking and user-account hacking is just going to run rampant, and I'm left wondering 'Where is my username and password actually stored?'. The answer, sadly, is wherever the dude who's running the instance/server is. In the 'Fediverse' your server instance might be hosted in a US or EU data center with proper digital and physical security, or it could be Joe Blows basement in Iowa running off a NAS. The easy-to-see future here is that Lemmy will fail to attract a critical mass of people because they'll initially arrive, after a few months their instances will just cease to exist/get shut down/the hosts will decide its no longer a fun hobby to do.
With a large corporation, they have the staff and resources to secure and maintain the servers physically and digitally, and keep staff up-to-date on current infosec threats and get out in front of them. Beyond that, if there IS a breach, they have the ability to recognize it, understand the legalities and requirements of reporting it, and can be held accountable by regulatory bodies. Joe doesn't have the resources to really maintain and keep a server running, nor the knowledge of his responsibilities for keeping the data safe digitally or physically.
On top of that, if Joe's basement loses power/gets hacked/Joe decides he's moving to San Fransisco and can't bring his NAS with him and the server goes down, and that's where my instance is hosted well there goes my entire account/comments/data.
-
Finding and subbing to communities is painfully difficult. It should be one-click, but somewhere I need to goto an external list, find what I want, and then copy/paste the URL into the search... and then 50% of the time, it doesn't work. This is an understandable growing pain and can likely be fixed by UI/UX upgrades, but for now it's a definite turn-off.
-
There simply is no content. I'm not a creator, I want content aggregated for me, and I've gotten used to having a single place to get it from that floods me with thousands of different articles/memes/posts/etc every minute. Until the user base arrives in one single place and starts generating content, there's no reason for most people like me to be there as by far the larger number of users never create anything at all and only exist to consume the content generated.
Thats not the point of being federated. It is very simple to understand once you get over the learning curve. The analogies to countries/states communicating to each other but being seperate helped me
See, this is yet another issue with Lemmy. This is not the post I replied to. I replied to https://lemmy.ca/post/636721, "Redditors, how do you like Lemmy?", and yet it posted it in a completely different topic!
This is very insightful, even though it ended up here :) thank you!
Sites don't store passwords, they store password hashes. There is no reason to give any personal info you aren't comfortable giving. You can use the site just fine without posting any
Hacking an account is still a valid concern though for various reasons , and hashes can still be used against password lists. Additionally, Two factor authentication is a necessity for sure. Now don’t get me wrong, I completely understand this feature is coming and that this is a developing service but many of these concerns do seem valid to me.
Hacking an account is still a valid concern though for various reasons
Let's assume you're doing the best practice thing and using a long and unique password for each service you use.
What benefit does a hacker have hacking your lemmy-based account? Considering that everything you post is public... There's simply nothing of value that you would obtain by "hacking" an account here... The only thing I can think of is if your a moderator of a community or an admin of an instance.
I just don't see any value to it... But even then.. 2fa is slated for v0.18 which is probably coming out in the next few weeks.
load more comments
(3 replies)
There needs to be ONE site, Lemmy.com, that people goto. This entire thing about having .whateveryouwant is VERY off putting.
The main feature of a federated system is that there's no one "owner" of the entire system; each admin uses their own servers and their own domain names. There's nothing stopping a server admin from using a .com domain name...
There’s no 2fa at all
There's basic TOTP 2FA now: https://github.com/LemmyNet/lemmy/issues/2363 but it hasn't been released yet (it's only available in the latest server beta). "Next-gen" TOTP (Webauthn/FIDO) is coming later: https://github.com/LemmyNet/lemmy/issues/3059
‘Where is my username and password actually stored?’. The answer, sadly, is wherever the dude who’s running the instance/server is. In the ‘Fediverse’ your server instance might be hosted in a US or EU data center with proper digital and physical security, or it could be Joe Blows basement in Iowa running off a NAS.
You should get to know your server admin, then. You have the freedom to pick any server you like :)
There simply is no content
Have you subscribed to many communities yet? You can browse other Lemmy servers to find interesting communities, and subscribe to them on your instance.
On a similar note, would it be possible to have something like "sharding" where one server has multiple synched copies on several people's machines? So lets say if one machine goes down for whatever reason there are others to still serve content? This could also help with distributing load across multiple machines so its less stressful on one, and we don't have situations such as whats going on with lemmy.ml now, where so many users are joining it that its frequently down and subscription statuses are stuck on pending
This is usually referred to as "high availability", where you'd have a hot failover to swap to in case the main server goes down. This is usually implemented with a load balancer that checks if the upstream server is alive before sending requests to it. If the upstream server isn't responding, switch to the other one.
A load balancer could also spread the load evenly across multiple machines, at least for reads. Generally there's far more reads than writes, and reads are more easily scalable since you can have database replicas that just need to sync in one direction.
I don't think Lemmy supports any of this yet though.
The other approach is to split the large instances into multiple smaller instances. For Fediverse stuff, I don't know which approach is considered "better".
A large open source project is generally more secure than a closed source one as it has more developers working on it and more security experts reading the code or testing it. If you want an example look at how many developers are working on Linux vs Windows and there security history. Once lemmy is big enough it has the potential to be more secure than things like Facebook or Twitter even.
You apparently don't understand how password hashes work. Nobody is storing your password on any well implemented website. You can't just grab someone's password if there is a breach, provided the people who built the website aren't completely incompetent.
With these misunderstandings and the fact federation was confusing to you I am not sure I believe your claim that you are a "tech-friendly" person. You sound like someone who has above average exposure to technology rather than someone who is actually skilled in it. There is nothing wrong with this in itself but please don't claim to be proficient when you aren't.
Curiously, I can't see lemmy.ml content in my Jerboa feed, but it's working in the browser.
Working for me (from UK).
EDIT: seems they migrated to a new server earlier today to handle their increased load. Maybe a hiccup related to that, or your DNS server is still pointing to the old instance or something (in which case it should eventually fix itself)
I get 502 in germany with mobile data and wifi
Are you on an Apple product, since that could be related to the “iCloud’s Private Relay” issue, since you're getting a 502 error?
i tried with pixel 7 and macbook but both didnt work. Private Relay is disabled.
load more comments
(2 replies)
I noticed that there's one thread on lemmy.ca, made by an account on lemmy.ml, that I just absolutely cannot reply to. It hangs, and nothing gets posted. I'm able to reply to other threads in the same community.
How exactly are these things structured? Like, could that issue be related to lemmy.ml doing its migration, even if the topic was posted on another instance? It was very weird behavior.
It's down for me too - southwestern U.S.
AWS Outage should be resolved now (it impacts my job, I would know).
IMO, looks like something else is up with this.
Down for me as well. Greece.
load more comments
(1 replies)
Works AOK from here in NL
I haven’t yet looked at the Lemmy architecture of the server, but it should be fairly straightforward to have a redundant region on AWS and use Route53 for the failover.
view more: next ›