Lemmy.World Announcements

28381 readers

1 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to [email protected] e-mail.

Report contact

DM https://lemmy.world/u/lwreport
Email [email protected] (PGP Supported)

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Join the team

founded 1 year ago

MODERATORS

[email protected]

2232

Lemmy.world status update 2023-07-04 (lemmy.world)

submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]

322 comments fedilink hide all child comments

Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won't work as desired.. we see several issues:

Performance

Loading is mostly OK, but sometimes things take forever
We (and you) see many 502 errors, resulting in empty pages etc.
System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
2FA seems to be a problem for many people. It doesn't always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @[email protected] is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 59 points 1 year ago (1 children)

@ruud > That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

Hmm, makes me curious if there is a Lemmy memory leak, or simply that the load wants to stabilize above of the RAM you have? I hope contributions can help you with another 32 GB RAM? Thank you for your work! 🍻

[–] [email protected] 63 points 1 year ago (2 children)

We have 128GB of RAM. It just skyrockets after a while!

[–] [email protected] 26 points 1 year ago

@ruud Oh damn. This spontaneously sounds crazy but I’m admittedly a novice at servers on this scale.

[–] [email protected] 5 points 1 year ago (1 children)

I haven't checked into the code yet, but I imagine you can map out what all is in memory and force more aggressive garbage collection to find some middle ground.

[–] [email protected] 13 points 1 year ago (1 children)

Rust is not a GC language.

[–] [email protected] 2 points 1 year ago

I read down the list afterwards and found it was using Rust. I skimmed through the source and it is well organized, but would still take quite a while to get up to speed on.

I saw unit and integration tests. It might be beneficial to generate or capture some data to replay to simulate the load and add debugging. I don't know much about the abstraction layers. I did see opentelemetry, which is a project I got frustrated with on the lack of stability (fast changes on api).

I have only dabbled with Rust to test the waters. The largest thing I've made was a GUI snake game, and made it portable so it could be compiled for cross platform.