Lemmy Server Performance

420 readers

1 users here now

lemmy_server uses the Diesel ORM that automatically generates SQL statements. There are serious performance problems in June and July 2023 preventing Lemmy from scaling. Topics include caching, PostgreSQL extensions for troubleshooting, Client/Server Code/SQL Data/server operator apps/sever operator API (performance and storage monitoring), etc.

founded 1 year ago

MODERATORS

[email protected]

lemmy_server API for Clients and Federation alike, concurrency self-awareness, load sheding, and self-tuning (lemmy.ml)

submitted 1 year ago by [email protected] to c/[email protected]

3 comments fedilink hide all child comments

Heavy loads have been brought up several times, major social events where people flock to social media. Such as a terrorist bombing, submarine sinking, earthquake, nuclear meltdown, celebrity airplane crash, etc.

Low-budget hosting to compete with the tech giants is also a general concern for the project. Trying not to have a server that uses tons of expensive resources during some peak usage.

Load shedding and self-tuning within the code and for SysOps to have some idea if their Lemmy server is nearing overload thresholds.

See comments:

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago)

Taking a step back, with the most basic data in the database... votes are by far the biggest concern.

I think a study of the lifetime of a vote in terms of I/O from a user pressing "upvote" and how it flows through the system, gets totaled, and how that is then reflected in output of content. Both postings and comments.

I'm always concerned about INSERT into the database locking I/O for logging and INDEX - so these are the most database actions. Second, they are also a caching problem, because if you try to provide real-time data on votes - even if the comments and postings are the same, a client has to go back to the database to get that most recent data. It's pretty, it's cool, but it's invaliding your cache.

It would be really nice if we can start to get an English understanding of the process.

I also think it would be ideal to have a setting to disable it from writing to the database, toss them somewhere else like a sequentially write-only text file or some SqLITE database - and have a "less overhead" install of Lemmy - and possibly as a fallback during high-concurrency concerns.