this post was submitted on 05 Jun 2023
34 points (97.2% liked)

Lemmy

12531 readers
17 users here now

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to [email protected].

founded 4 years ago
MODERATORS
 

With forewarning about a huge influx of users, you know Lemmy.ml will go down. Even if people go to https://join-lemmy.org/instances and disperse among the great instances there, the servers will go down.

Ruqqus had this issue too. Every time there was a mass exodus from Reddit, Ruqqus would go down, and hardly reap the rewards.

Even if it's not sustainable, just for one month, I'd like to see Lemmy.ml drastically boost their server power. If we can raise money as a community, what kind of server could we get for 100$? 500$? 1,000$?

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 12 points 1 year ago (7 children)

The site currently runs on the biggest VPS which is available on OVH. Upgrading further would probably require migrating to a dedicated server, which would mean some downtime. Im not sure if its worth the trouble, anyway the site will go down sooner or later if millions of Reddit users try to join.

[–] [email protected] 11 points 1 year ago

There will either be an hour of downtime to migrate and grow or days of downtime to fizzle.

I love that there's an influx of volunteers, including SQL experts, to mitigate scaling issues for the entire fediverse but those improvements won't be ready in time. Things are overloading already and there's less than a week before things increase 1,000-fold, maybe more.

[–] [email protected] 6 points 1 year ago (11 children)
8 vCore
32 GB RAM

😬

2 follow-ups:

  • Can we replace Lemmy.ml with Join-lemmy.org when Lemmy.ml is overloaded/down?
  • Does LemmyNet have any plans on being Kubernetes (or similar horizontal scaling techniques) compatible?
[–] [email protected] 2 points 1 year ago (1 children)

Maybe some dns fail-over for lemmy.ml to point to join-lemmy.org might be cool indeed 🤔

[–] [email protected] 1 points 1 year ago

Yeah, was thinking of a DNS based solution as well. Probably the easiest and most effective way to do it?

[–] [email protected] 2 points 1 year ago

We need Self hosted team and team networking to represent. Would be amazing to see some community support in scaling Lemmy up.

load more comments (9 replies)
[–] [email protected] 5 points 1 year ago* (last edited 1 year ago) (1 children)

I'm sure you know this, but getting progressively larger servers is not the only way, why not scale horizontally?

I say this as someone with next to no idea how Lemmy works.

[–] [email protected] 16 points 1 year ago (1 children)

Its better to optimize the code so that all instances benefit.

[–] [email protected] 3 points 1 year ago* (last edited 1 year ago) (1 children)

Is it possible to make Lemmy (the system as a whole) able to be compatible with horizontally scaling instances? I don't see why an instance has to be confined to one server, and this would allow for very large instances that can scale to meet demand.

Edit: just seen your other comment https://lemmy.ml/comment/453391

[–] [email protected] 4 points 1 year ago (3 children)

It should be easy once websocket is removed. Sharded postgres and multiple instances of frontend/backend. Though I don't have any experience with this myself.

[–] [email protected] 5 points 1 year ago (1 children)

There is already a docker image so that should not be too hard. I'd be happy to set something up, but (as others have said) the DB will hit a bottleneck relatively quickly.

I like the idea of splitting off the image processing.

[–] [email protected] 1 points 1 year ago (1 children)

Image processing isnt causing any noticable cpu load.

[–] [email protected] 1 points 1 year ago (1 children)

I saw someone say it was, obviously I have no access to data.

[–] [email protected] 1 points 1 year ago

Maybe on another instance but not on lemmy.ml

[–] [email protected] 3 points 1 year ago

I think that is unavoidable, Look at the most popular subreddits , they can get something like 80 million upvotes and 66K comments per day, do you think a single server can handle that?

Splitting communities just so that it will be easier technically is not good UX.

[–] [email protected] 3 points 1 year ago

@nutomic @Lobstronomosity In one of the comments I thought I saw that the biggest CPU load was due to image resizing.

I think it might be easier to split the image resizer off to its own worker that can run independently on one (or more) external instances. Example: client uses API to get a temporary access token for upload, client uploads to one of many image resizers instead of the main API, image resizer sends output back the main API.

Then your main instance never sees the original image

[–] [email protected] 1 points 1 year ago (1 children)

Do you have the frontend a DB serving in the same VPS? If so it would be a great time to split them. Likewise if you DB is running in a VPS, you're likely suffering from significant steal from the hypervisor so you would benefit from switching to a dedicated box. My API calls saw a speedup of 10x just from switching from a VPS DB to a Dedicated Box DB.

I just checked OVH VPS offers and they're shit! Even at 70 Eur dedicated on hetzner, you would gain more than double those resources without steal. I would recommend switching your DB ASAP for immediate massive gains.

If you're wondering why you should listen to me, I built and run https://aihorde.net and are handling about 5K concurrent connections currently.

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (1 children)

Hetzner is very strict about piracy so thats not an option. And now is almost weekend so I wont have time for a migration. Anyway there are plenty of other instances in case lemmy.ml goes down.

Edit: I also wouldnt know which size of dedicated server to choose. No matter what I pick, it will get overloaded again after a week or two.

[–] [email protected] 1 points 1 year ago

Even if you choose Hetzner, it won't even know it has anything to do with piracy because it will be just hosting the DB, and nobody will know where your DB is. That fear is overblown.

Likewise believe me a dedicated server is night and day from a VPS.

[–] [email protected] 0 points 1 year ago

Is it running in a single docker container or is it spread out across multiple containers? Maybe with docker-machine or kubernetes with horizontal scaling, it could absorb users without issue - well, except maybe cost. OVH has managed kubernetes.

[–] [email protected] 0 points 1 year ago (1 children)

So reading this correctly, it's currently a hosting bill of 30 Euro a month?

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (1 children)

No, thats the 8 GB memory option... if its the biggest, it should be around 112 €. Meanwhile i keep wondering if i should let Lemmy stay on the current KVM (which is similarely specked but with dedicated cores and stuff) or if it is better to move it to one of my dedis just in case... well... will see xD

[–] [email protected] 1 points 1 year ago (5 children)

Its the one for 30 euros, Im not seeing any vps for 112. Maybe thats a different type of vps?

[–] [email protected] 0 points 1 year ago (5 children)

in vservers, it depends on the memory … and storage option for the one starting at 30…

load more comments (5 replies)
load more comments (4 replies)
[–] [email protected] 0 points 1 year ago (3 children)

What's the current bottleneck?

[–] [email protected] 6 points 1 year ago (1 children)

SQL. We desperately need SQL experts. It's been just me for yeRs, and my SQL skills are pretty terrible.

[–] [email protected] 2 points 1 year ago

Put the whole DB in RAM :-)

Makes me remember optimization, lots of EXPLAIN and JOIN pain, on my old MySQL multiplayer game server lol. A shame I'm not an expert ...

[–] [email protected] 2 points 1 year ago (4 children)

There are some SQL database optimisations being discussed right now and apparently the picture resizing on upload can be quite CPU heavy.

[–] [email protected] 3 points 1 year ago (1 children)

I've been helping on the SQL github issue. And I think the biggest performance boost would be to separate the application and postgresql onto different servers. Maybe even use a hosted postgresql temporarily, so you can scale the db at the press of a button. The app itself appears to be negligible in terms of requirements (except the picture resizing - which can also be offloaded).

But running a dedicated db on a dedicated server - as close to the bare metal as possible give by far the best performance. And increase it for more connections. Our production database at my data analytics startup runs a postgresql instance on an i9 server with 16 cores, 128GB RAM, and a fast SSD. We have 50 connections set up, and the run pgbouncer to allow up to 500 connections to share those 50. And it seamlessly runs heavy reporting and dashboards for more than 500 business customers with billions of rows of data. And costs us less than US$200pm at https://www.tailormadeservers.com/.

[–] [email protected] 1 points 1 year ago

And I think the biggest performance boost would be to separate the application and postgresql onto different servers.

I think hexbear.net (an older lemmy fork-ed site) is working on this in conjunction with moving back to a modern lemmy version

[–] [email protected] 3 points 1 year ago (2 children)

SQL dev here. Happy to help if you can point me in the direction of said conversation. My expertise is more in ETL processes for building DWs and migrating systems, but maybe I can help?

load more comments (2 replies)
[–] [email protected] 1 points 1 year ago (1 children)

And may be the bandwidth. Serve thousands and thousands need at minimum 1gbps.

[–] [email protected] 3 points 1 year ago

Its mostly text so bandwidth shouldnt be a problem.