Hey folks!

Unfortunately, roughly 2 hours ago, lemm.ee went offline. The cause was our load balancer: it suddenly decided that all of our servers had become unhealthy, despite all health checks responding successfully when I requested them directly. In such cases, the load balancer stops serving all requests, effectively meaning that lemm.ee is unreachable for all users. I am still not sure what exactly caused the issue, but I will try to investigate more over the weekend.

For now, we have partially recovered, and I am continuing to work on remaining issues. Hopefully we will be back to 100% very soon. Sorry for the inconvenience!

Hey folks

Just a heads up that I will be doing some minor database maintenance shortly. I expect the downtime to last <5 minutes.

Have a nice day!

Update: maintenance is complete!

Hey all!

Upcoming lemm.ee cakeday

Can you believe that lemm.ee is almost 1 year old? In just a couple of weeks (specifically, on the 9th of June), we will be able to celebrate our first instance cakeday.

I am thinking of compiling some stats about how lemm.ee has been used in its first year, if you have any specific stats in particular you would like to see, feel free to comment below. I will try to accommodate any ideas as I start gathering this info!

Infrastructure updates

A few weeks ago, I posted about plans to make some changes to our infrastructure in order to deal with different intermittent networking issues.. It took a bit longer than I hoped (just did not manage to get enough free time between then and now), but I am happy to report that this work has now been completed! Additionally, I have decommissioned our stand-alone pict-rs server.

With the two changes mentioned above, I believe lemm.ee should now be much more resilient going forwad, and I expect a significantly lower rate of infrastructure-related issues for the rest of the year!

I'll leave a tehcnical overview about the problem & solution below for those interested, but if these details don't interest you, then you can safely skip the rest of this post.

For context, lemm.ee has been hosted on Hetzner servers for most of this year (having migrated from DigitalOcean initially), with everything except our database being hosted on the Hetzner Cloud side, and the database itself living on a powerful dedicated Hetzner server. This mix allows a great amount of flexibility for redeploying and horizontally scaling our application servers, while still allowing a really cost-effective way of hosting a quite resource-hungry database.

In order to facilitate networking between the cloud servers and the dedicated database server (which live in different networks), Hetzner provides a service named "vSwitch". This service basically allows you to connect different servers together in a private network. Unfortunately, I discovered quite quickly that this service is very unreliable. During the short few months that we have been using the vSwitch, we have gone through one extended period of downtime (where the service was just completely broken for several hours), as well as dozens (if not hundreds at this point) intermittent disconnects, where servers randomly lose their connections over the vSwitch. After such a disconnect, the connection never recovers without manual intervetion.

For most lemm.ee users, the majority of these vSwitch issues have been mostly invisible, as we have redundancy in our servers - if one server loses its connection to the database, other servers will take over the load. Additionally, I have generally been able to respond quite quickly to issues by redeploying the broken servers (or deploying other temporary workarounds). However, in addition to a huge amount of these issues which lemm.ee users hopefully haven't ever noticed, there have also been a few short periods of downtime this year so far, as well as a few cases of federation delays. These more extreme cases were generally caused by multiple servers losing their vSwitch connections at the same time.

After several attempts to work around these issues, I decided that we need to migrate away from vSwitch.

As of earlier today, lemm.ee is no longer using Hetzner's vSwitch at all!

I finally found enough time earlier today to focus on this migration, and I was able to successfully complete it. None of our networking is relying on the vSwitch anymore.

In the end, I went with quite a simple solution - I configured a host-level firewall (nftables) on our database dedicated server, which will deny all connections by default. Whenever any cloud servers are added/removed, their corresponding public IP addresses are added/removed in the allowlist of our database firewall. It would have been ideal to do this whole logic in Hetzner's own firewall, but that one unfortunately has a limit of only 10 rules per server, which is just not enough for our setup.

Bonus: our pict-rs server has been decommissioned!

Pict-rs is the software which Lemmy uses for everything related to media (image storage mostly). Initially, pict-rs required a local filesystem to store both files as well as metadata about files. Since the beginning, lemm.ee has used a dedicated server just for pict-rs, in order to ensure we could easily redeploy the rest of our servers without losing any images.

Over the past year, pict-rs has gained the ability to store files in object storage, and metadata in a PostgreSQL database. This meant that the server running pict-rs itself no longer contained any of the important data, so it became possible to redeploy without losing any images. Additionally, this meant that it would be possible to run multiple pict-rs servers in parallel.

While we had already migrated our pict-rs server to use object storage and PostgreSQL several months ago, we still had the single dedicated pict-rs server up until today. I have been planning for a while to decommission this server, and start running pict-rs directly on each one of our Lemmy application servers. Earlier today, I was able to complete this plan. This should hopefully mean that our pict-rs server is less likely to get overloaded, and it also means a tiny reduction in our overall monthly infrastructure bill (due to one less server running).

With the above changes, I think our infrastructure has become more robust, and hopefully, we will experience less issues with images, federation, and general downtime going forward.

That's all from me for now. Feel free to leave any thoughts or questions in the comments, and as always, I hope you're having a great day!

Hey folks!

This is a quick notice about a change to our moderation policy.

We have had a policy on lemm.ee for administration and federation nearly since the very beginning. This policy has also always included a section about moderator responsibilities. Today, we have made two changes to this policy:

1. The policy has been renamed to Policy for administration, moderation, federation - this is to make it clear that the policy is also relevant for mods
2. We have introduced a new responsibility for moderators, they must "Ensure that they only provide accurate and clear reasons for mod actions".

The reason for the addition is that mod log actions federate out to other instances, and are more or less permanent (due to how Lemmy and federation works right now). This means that users do not really currently have any easy way to clarify or defend themselves against inaccurate accusations in the mod log.

As always, I am very grateful to all mods for your efforts in building awesome communities on lemm.ee. I hope you can understand why this new policy is necessary - I do not want to make your lives more difficult, the goal is to just try and reduce any mod log related misunderstandings in the future.

Thank you for reading and have a nice day!

Hey folks!

We unfortunately had about half an hour of unplanned downtime today. This was caused by an issue with our hosting provider. The issue is solved for now, and I am planning to make some changes to prevent similar issues in the future. Sorry for the inconvenience!

Technical details

Our servers are communicating with our database over Hetzner's "vSwitch" service. Unfortunately, this service seems to be quite flaky - over the past few months, I have had to deal with the connection just dropping without recovering many times. Mostly this has not resulted in any noticeable downtime, as we have redundant servers, so even if one of them stops working, it won't affect lemm.ee users. However, in this instance, all of our API servers lost their connection to our database at the same time, which resulted in actual downtime.

I have now decided to migrate our setup away from the vSwitch in the near future to hopefully stop these issues for good. Should be possible to do this migration without any downtime, I just need to set aside some time to actually create an alternative solution for us, most likely over the coming weekend. I will update this post once the migration is complete.

Update: the migration is now complete! You can read more here.

Hey folks!

I've been steadily working through the roadmap for lemmy-ui-next (which is a new alternative Lemmy frontend), and it's getting to a point where I think https://next.lemm.ee is becoming quite usable. I've been personally using it as my main Lemmy frontend for several weeks now, and I know there are a few other brave users doing the same, so at this point, I'm confident enough to ask the wider lemm.ee population to try it out and share some honest feedback.

If you're at all interested in this project, I would massively appreciate it if you could spend some time using https://next.lemm.ee and letting me know how you feel about it. I'm interested to hear about things like:

• are you running into any issues or bugs
• are there any things that generally annoy you
• are you missing any features
• what would it take for lemmy-ui-next to become your preferred frontend
• anything else that comes to mind

Please keep in mind that this is still a work in progress - some features are planned but not implemented yet (see the roadmap linked above for more details), other features are half-finished and may be a bit buggy still!

Any feedback would really help me out, so please don't hesitate to share!

Hey folks

This is just a quick heads up that I need to perform some maintenance & upgrades on our database server, which unfortunately will require downtime. I don't expect the downtime to last for longer than 2-3 minutes, but just wanted to give a heads up first so you know not to be concerned.

That's all, hope you have a great week!

Edit: maintenance complete!

Hello, world!

Edit: first test edit!

Milestone 1 complete!

This is just a mini-announcement & celebration for the fact that I have completed the scope for the first milestone I set for myself in the roadmap.

Of course, I am still planning to keep improving and tweaking things as I go, but in terms of the raw list of features, the work for milestone 1 is complete. I am now going to take a day or two to clean up the code and work on some performance optimizations, and then in the later half of the week, I will continue working towards milestone 2, starting with commenting features!

If anybody is interested (and brave), please feel free to check it out at https://next.lemm.ee, and feel free to share any thoughts and feedback in the comments!

Hello, friends!

TL;DR: I am working on a new Lemmy frontend in nextJS. There is still much work to be done, but you can already have an early look at https://next.lemm.ee

First of all, quick note to lemm.ee users: I am making this announcement post in [email protected], as this is also a notice that I will be hosting an alternative frontend (lemmy-ui-next) for the first time on lemm.ee. Going forward, I will post updates about lemmy-ui-next in a separate dedicated community: [email protected]. If you're interested in future updates, please subscribe there!

What is lemmy-ui-next?

Lemmy is generally accessed through some kind of frontend UI. By default, Lemmy provides its own web interface (lemmy-ui), which you can find on the front page of most Lemmy instances (including lemm.ee). There are also several other independent frontends, for both the web and different mobile platforms, which I'm sure many of you are familiar with.

Lemmy-ui-next is a brand new alternative frontend, built from the ground up with modern and popular tooling - a framework known as NextJS. Lemmy-ui-next has (or aims to have) the following high-level features:

• Open source (AGPL)
• Drop-in replacement for lemmy-ui - same exact URL structure, so all existing links will continue working
• Very plain & minimalistic UI, strongly inspired by other link aggregator sites (of course including the original lemmy-ui!)
• Very basic and "typical" NextJS architecture, to encourage open source contributions
• Fully functional even when JavaScript is disabled (but works better with JS enabled!)
• Optimized data transfer between your browser and the server (filtering out only relevant data from the Lemmy API, caching, memoization)
• Strong focus on privacy and security (all authentication with the Lemmy API is done through secure httpOnly cookies, user IP addresses are not leaked to external image hosts, etc)

What is the current status of lemmy-ui-next?

I have mentally split the initial work I want to complete into 3 milestones:

1. Lurk - All read-only features of Lemmy
2. Participate - Voting/posting/commenting/DMs/reports, etc
3. Moderate - Handling reports, creating & managing communities, etc

I am now nearing completion of the first milestone. It's not 100% there yet, but you can already log in, browse, subscribe to communities and even vote. Some things may still look a bit wonky, and some features are still missing, but the core experience is getting there.

In terms of code contributions, I would ask anybody who is interested in getting involved to contact me first before working on anything. I am not looking for PRs just yet - the code structure is still a bit loose, and I am redefining it as I add more stuff. I would ideally really like to complete the first 3 milestones before opening things up for external contributors.

Who can use lemmy-ui-next?

At the moment, it is only hosted on this instance, at https://next.lemm.ee. I do not yet have any formal instructions for running it on other instances, but generally speaking, it is a simple NextJS app - to deploy it, you just need to do: npm install, npm run build and LEMMY_BACKEND=https://<your lemmy api here> npm run start.

Why not just improve lemmy-ui instead?

Lemmy-ui is an extremely important and valuable project. There has been a significant amount of hard effort put into it so far, and nobody can refute that it is the frontend which has really carried Lemmy to this point.

Unfortunately, there are some architectural problems with lemmy-ui (mostly related to how data is fetched and how sessions are stored in memory), all of which would require quite a significant rewrite to fix. Additionally, I think that the core technical solution used for lemmy-ui is just a bit too obscure, which has been an obstacle to my own contributions, as well as to contributions by others. If a rewrite is on the table anyway, then I believe a different technology is the best way forward.

Why not work on lemmy-ui-leptos instead?

Lemmy-ui-leptos is another rewrite of lemmy-ui, which is being lead by Lemmy maintainers. It is based around a Rust web framework called Leptos. I think this is really cool tech, and will be happy to host lemmy-ui-leptos on lemm.ee in the future as well.

There are a two key reasons why I personally decided to start working on another alternative, though:

• I have heard from several people on Lemmy that they feel like Leptos is a big barrier to entry in terms of them contributing
• Even for myself personally, I am very comfortable (and think I can move very fast) when working on something like NextJS, but with Leptos, I think the learning curve would be quite big and I would get much less done with any time I invest into it

My hope is that by providing a very vanilla alternative, I can provide an outlet for potential open source contributors who would like to work on Lemmy, but aren't prepared to do it with Leptos.

Does this mean that lemm.ee will change in ways I don't like?

First, let me be clear: lemm.ee will always host the default Lemmy frontend. This means lemmy-ui for now, and most likely lemmy-ui-leptos in the future.

I am however considering the possibility of switching things around at some point in the future, so that lemmy-ui-next will be hosted directly on lemm.ee, and lemmy-ui will be accessible on a different subdomain (like ui.lemm.ee). This would only happen once I have completed all 3 milestones for lemmy-ui-next. The main reason I am considering this is that I feel like I will always be in the best position to offer technical support to users on the frontend which I am myself maintaining. If you have any thoughts about this potential change, please let me know in the comments below!

That's about it for now!

This is something I've been thinking of doing for a while now, and I'm very excited to finally get the ball rolling! If you have a chance, please feel free to check out what https://next.lemm.ee looks like so far, and please let me know if you have any thoughts or feedback!

Hey

This is just a quick heads up that our host, Hetzner, has been experiencing networking issues today, which has caused some downtime for lemm.ee.

I have a workaround in place for now, so we should (fingers crossed) be recovering at the moment, but I am still waiting on the proper solution from Hetzner. You can track their issue here: https://status.hetzner.com/incident/9406c500-9c8b-48be-9591-a73691134096

Also, this is a good opportunity to remind everybody about https://status.lemm.ee - you can be sure that I will provide updates on that page as soon as I am aware of & dealing with any issues. I have been posting status updates for the current issue there as well.

Sorry for the inconvenience and I hope you have an otherwise great day!

UPDATE: Hetzner claims they have fixed the issue, but the problems have not been resolved for lemm.ee servers yet, so I am keeping my temporary workaround active for now. Will continue troubleshooting this tomorrow.

UPDATE 2: Hetzner has now fixed their issue, and our network has been restored to its original optimized state.