Lemmy Server Performance

420 readers

1 users here now

lemmy_server uses the Diesel ORM that automatically generates SQL statements. There are serious performance problems in June and July 2023 preventing Lemmy from scaling. Topics include caching, PostgreSQL extensions for troubleshooting, Client/Server Code/SQL Data/server operator apps/sever operator API (performance and storage monitoring), etc.

founded 1 year ago

MODERATORS

[email protected]

lemmy_server API for Clients and Federation alike, concurrency self-awareness, load sheding, and self-tuning (lemmy.ml)

submitted 1 year ago by [email protected] to c/[email protected]

3 comments fedilink hide all child comments

Heavy loads have been brought up several times, major social events where people flock to social media. Such as a terrorist bombing, submarine sinking, earthquake, nuclear meltdown, celebrity airplane crash, etc.

Low-budget hosting to compete with the tech giants is also a general concern for the project. Trying not to have a server that uses tons of expensive resources during some peak usage.

Load shedding and self-tuning within the code and for SysOps to have some idea if their Lemmy server is nearing overload thresholds.

See comments:

top 2 comments

sorted by: hot top controversial new old

[–] [email protected] 2 points 1 year ago

10 second http timeout

https://github.com/LemmyNet/lemmy/blob/f5209fffc1de527db7ea007d463c158b36fda515/src/lib.rs#L39

I almost worry this default 10 second timeout is too low, and that is actually causing federation retries to flock between major peer servers. But this is the kind of self-tuning within the app that self-awareness of concurrency could adjust.

Areas I know of that Lemmy does HTTP:

Federation replication of content to peers
When making a Post on Lemmy, a url posting fetches a preview of the content

I'm not even sure where that value is used, any help on delving into this is appreciated.

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago)

Taking a step back, with the most basic data in the database... votes are by far the biggest concern.

I think a study of the lifetime of a vote in terms of I/O from a user pressing "upvote" and how it flows through the system, gets totaled, and how that is then reflected in output of content. Both postings and comments.

I'm always concerned about INSERT into the database locking I/O for logging and INDEX - so these are the most database actions. Second, they are also a caching problem, because if you try to provide real-time data on votes - even if the comments and postings are the same, a client has to go back to the database to get that most recent data. It's pretty, it's cool, but it's invaliding your cache.

It would be really nice if we can start to get an English understanding of the process.

I also think it would be ideal to have a setting to disable it from writing to the database, toss them somewhere else like a sequentially write-only text file or some SqLITE database - and have a "less overhead" install of Lemmy - and possibly as a fallback during high-concurrency concerns.

load more comments