Lemmy Server Performance

420 readers

1 users here now

lemmy_server uses the Diesel ORM that automatically generates SQL statements. There are serious performance problems in June and July 2023 preventing Lemmy from scaling. Topics include caching, PostgreSQL extensions for troubleshooting, Client/Server Code/SQL Data/server operator apps/sever operator API (performance and storage monitoring), etc.

founded 1 year ago

MODERATORS

[email protected]

Redis, Memcached, dragonfly for Lemmy and 2010 presentation on scaling.... (lemm.ee)

submitted 1 year ago by [email protected] to c/[email protected]

3 comments fedilink hide all child comments

Lemmy is incredibly unique in it's stance of not using Redis, Memcached, dragonfly... something. And all the CPU cores and RAM for what this week is reported as 57K active users across over 1200 Instance servers.

Why no Redis, Memcached, dragonfly? These are staples of API for scaling.

Anyway, Reddit too started with PostgreSQL and was open source.

MONDAY, MAY 17, 2010

http://highscalability.com/blog/2010/5/17/7-lessons-learned-while-building-reddit-to-270-million-page.html

"and growing Reddit to 7.5 million users per month"

Lesson 5: Memcache
The essence of this lesson is: memcache everything.

They store everything in memcache: 1. Database data 2. Session data 3. Rendered pages 4. Memoizing (remember previously calculated results) internal functions 5. Rate-limiting user actions, crawlers 6. Storing pre-computing listings/pages 7. Global locking.

They store more data now in Memcachedb than Postgres. It’s like memcache but stores to disk. Very fast. All queries are generated by same piece of control and is cached in memcached. Change password Links and associated state are cached for 20 minutes or so. Same for Captchas. Used for links they don’t want to store forever.

They built memoization into their framework. Results that are calculated are also cached: normalized pages, listings, everything.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago)

They store more data now in Memcachedb than Postgres. It’s like memcache but stores to disk. Very fast.

And that was 7.5 million active users with 2010 hard drive technology. Hardware in 2023 is way faster. Lemmy is incredibly unique in it’s stance of not using Redis, Memcached, dragonfly… something. And all the CPU cores ... and still crashing.

For 4.5 year old code, I think the crashing speaks for itself. But this is the days of Elon Musk and Reddit changing their API with 1 month notice... the social aspect of social media seems to be making choices that just seem ... odd. And it isn't just Twitter, it isn't just Reddit...

I mean, May 2023 people flocking to the Lemmy door, your servers can't keep up with the poor-performing SQL... add scaling tools like like Redis, Memcached, dragonfly to the platform. There was a 40-day countdown to the Reddit API change.

I don't understand what has happened here on Lemmy since May any more than what Elon Musk is doing to Twitter to X. It's bewildering. Redis, Memcached, dragonfly are easily integrated...

"Redis has been a staple of the web ecosystem for years. It’s often used for caching, as a message broker, or simply as a database. In this guide, we’ll demonstrate how to use Redis inside a Rust web application.".... and the same exists for Rust and memcached, and Rust and dragonfly... Lemmy was in active development for 4 years, it's impossible to accept that nobody knows of tools like dragonfly and Redis.... its some kind of social choice at play, one that I find bewildering and frankly, has put me into tears as people seem to think the server crashes can't be addressed!...

the crashing servers speaks for itself, but I guess I spoke for it too. It's been a sad journey since May :( I don't understand what is gong on in social media audiences that Elon Musk and Reddit and Lemmy are making such choices.