General Lemmy.Cafe

Everything went through smoothly, but please do report if you notice anything.

Lemmy Cafe will be having its database upgraded.

Reasons

• PostgreSQL 17 has been released and the changelog is promising a lot of IO improvements. Lemmy sure could use it given the constant stream of small events flowing in.

Plan

• Point nginx to the maintenance page
• Shut down PostgreSQL 16
• Run the upgrade tool
• Start up PostgreSQL 17
• Point nginx to lemmy

Expected downtime

About an hour, if things go well. More if not so.

Will try to keep the maintenance page updated.

Here's the timezone converter.

Due to the recent @Soup's post I have decided to do some housekeeping. I've been getting frustrated at lemmy's performance at times as well and this simply was a wake up call, if you will.

I am sorry about no advance downtime annoucement - Sunday is the only day I can really put any meaningful amount of time into lemmy.

---

Things done today:

• Upgraded database VM 1 core 2GB -> 2 cores 4GB. Double the compute, double the memory.
• Adjusted database config to account for increased resources
• Adjusted hugepages config to account for increased database's requests
• Updated both lemmy and database VMs
• Rebooted the lot

---

Thank you for your patience. I will also use this moment of focus to write up a financial report in a separate post.

Also, thank you, @Soup!

---

EDIT: I've also marked all instances known to lemmy.cafe as active. What this means is that lemmy.cafe will now keep retrying to federate to everything very aggressively. This is a compute-intensive process and will also impact performance for a few hours, until exponential backoff kicks in. I've done it to revive any falsely-marked-as-dead instances; there's no fix for it on lemmy itself.

Due to the recent @Soup's post I have decided to do some housekeeping. I've been getting frustrated at lemmy's performance at times as well and this simply was a wake up call, if you will.

I am sorry about no advance downtime annoucement - Sunday is the only day I can really put any meaningful amount of time into lemmy.

---

Things done today:

• Upgraded database VM 1 core 2GB -> 2 cores 4GB. Double the compute, double the memory.
• Adjusted database config to account for increased resources
• Adjusted hugepages config to account for increased database's requests
• Updated both lemmy and database VMs
• Rebooted the lot

---

Thank you for your patience. I will also use this moment of focus to write up a financial report in a separate post.

Also, thank you, @Soup!

---

EDIT: I've also marked all instances known to lemmy.cafe as active. What this means is that lemmy.cafe will now keep retrying to federate to everything very aggressively. This is a compute-intensive process and will also impact performance for a few hours, until exponential backoff kicks in. I've done it to revive any falsely-marked-as-dead instances; there's no fix for it on lemmy itself.

Everthing about this instance crawls.

Everthing about this instance crawls.

We're now running on dual thread 4 gigs or ram. Hopefully that alleviates some recent instability.

Database is on a separate VM, seems to be holding up so far.

We're now running on dual thread 4 gigs or ram. Hopefully that alleviates some recent instability.

Database is on a separate VM, seems to be holding up so far.

This was a bugfix release.

The [email protected] has been reverted to being public as the private switch appears to break notifications.

Due to increased compute demand of lemmy software the VM lemmy.cafe is running on had to become a little beefier. DB VM is still chugging along without changes.

Happy socials!

---

As always - any financial help is greatly appreciated! To donate click the banner here or in the sidebar:

❤, Illecors

This was a bugfix release.

The [email protected] has been reverted to being public as the private switch appears to break notifications.

Due to increased compute demand of lemmy software the VM lemmy.cafe is running on had to become a little beefier. DB VM is still chugging along without changes.

Happy socials!

---

As always - any financial help is greatly appreciated! To donate click the banner here or in the sidebar:

❤, Illecors

Timeout uploading images, either for posts or comments. Tried Lemmy-UI, Voyager and Photon.

Timeout uploading images, either for posts or comments. Tried Lemmy-UI, Voyager and Photon.

This is a big release - the most interesting, probably, being local [non-federated] communities. There are a lot of other features, as well, so don't hesitate reading the release announcement!

As a first attempt, I have made [email protected] a local community. This post should not federate anywhere else.

The upgrade itself was a bit rocky, and, admittedly, caused by the setup that has strayed off the official guidelines. It has, however, helped in troubleshooting countless times previously, so it's not going away. More care will simply have to be taken next time.

For those curious - the bit that caused all the trouble was having separate users to connect to the database for each of backend, federation and tasks containers. Something changed where a new schema was created/started getting used and tasks container did not have access to said schema. As such, it could not finish the migration scripts.

As always - any financial help is greatly appreciated! To donate click the banner here or in the sidebar:

❤, Illecors

This is a big release - the most interesting, probably, being local [non-federated] communities. There are a lot of other features, as well, so don't hesitate reading the release announcement!

As a first attempt, I have made [email protected] a local community. This post should not federate anywhere else.

The upgrade itself was a bit rocky, and, admittedly, caused by the setup that has strayed off the official guidelines. It has, however, helped in troubleshooting countless times previously, so it's not going away. More care will simply have to be taken next time.

For those curious - the bit that caused all the trouble was having separate users to connect to the database for each of backend, federation and tasks containers. Something changed where a new schema was created/started getting used and tasks container did not have access to said schema. As such, it could not finish the migration scripts.

As always - any financial help is greatly appreciated! To donate click the banner here or in the sidebar:

❤, Illecors

example

unless there is an intentional reason for this, would love to see this working as i value the conversations going on over there :)

example

unless there is an intentional reason for this, would love to see this working as i value the conversations going on over there :)

It's been a while since I've last done one of those. Life happened, but things are settling down slowly. I'll combine all five months since September into a single post, but will keep them separated visually.

Income is applied on date of arrival. Stripe (LiberaPay uses it as processor) pays out in advance and as such income shows up as 0, even though LiberaPay banner shows a non-zero value.

Any financial help is greatly appreciated! To donate click the banner here or in the sidebar:

February 2024

Contributions

• LiberaPay: £0

Expenses

• Servers: £38.94

Month balance

• -£38.94

Previous balance

• -£216.05

Balance to date

• -£254.99

---

January 2024

Contributions

• LiberaPay: £0

Expenses

• Servers: £38.66

Month balance

• -£38.66

Previous balance

• -£177.39

Balance to date

• -£216.05

---

December 2023

Contributions

• LiberaPay: £0

Expenses

• Servers: £38.59
• Backblaze B2: £0.64

Month balance

• -£39.23

Previous balance

• -£138.16

Balance to date

• -£177.39

---

November 2023

Contributions

• LiberaPay: £0

Expenses

• Servers: £38.74

Month balance

• -£38.74

Previous balance

• -£99.42

Balance to date

• -£138.16

---

October 2023

Contributions

• LiberaPay: £0

Expenses

• Servers: £37.32

Month balance

• -£37.32

Previous balance

• -£62.1

Balance to date

• -£99.42

---

Finance History

	September 2023	August 2023	July 2023	June 2023
Contributions	£11.41	£15.26	£0	£0
Expenses	£28.85	£24.75	£19.18	£15.99
Difference	-£17.44	-£9.49	-£19.18	-£15.99
Balance	-£62.1	-£44.66	-£35.17	-£15.99

---

Previous reports

---

September 2023

---

August 2023

---

July 2023

---

June 2023

It's been a while since I've last done one of those. Life happened, but things are settling down slowly. I'll combine all five months since September into a single post, but will keep them separated visually.

Income is applied on date of arrival. Stripe (LiberaPay uses it as processor) pays out in advance and as such income shows up as 0, even though LiberaPay banner shows a non-zero value.

Any financial help is greatly appreciated! To donate click the banner here or in the sidebar:

February 2024

Contributions

• LiberaPay: £0

Expenses

• Servers: £38.94

Month balance

• -£38.94

Previous balance

• -£216.05

Balance to date

• -£254.99

---

January 2024

Contributions

• LiberaPay: £0

Expenses

• Servers: £38.66

Month balance

• -£38.66

Previous balance

• -£177.39

Balance to date

• -£216.05

---

December 2023

Contributions

• LiberaPay: £0

Expenses

• Servers: £38.59
• Backblaze B2: £0.64

Month balance

• -£39.23

Previous balance

• -£138.16

Balance to date

• -£177.39

---

November 2023

Contributions

• LiberaPay: £0

Expenses

• Servers: £38.74

Month balance

• -£38.74

Previous balance

• -£99.42

Balance to date

• -£138.16

---

October 2023

Contributions

• LiberaPay: £0

Expenses

• Servers: £37.32

Month balance

• -£37.32

Previous balance

• -£62.1

Balance to date

• -£99.42

---

Finance History

	September 2023	August 2023	July 2023	June 2023
Contributions	£11.41	£15.26	£0	£0
Expenses	£28.85	£24.75	£19.18	£15.99
Difference	-£17.44	-£9.49	-£19.18	-£15.99
Balance	-£62.1	-£44.66	-£35.17	-£15.99

---

Previous reports

---

September 2023

---

August 2023

---

July 2023

---

June 2023

due to the offending user being banned from the federating instance before the post was removed? example

edit: i recognize this is not a local issue and may be more of a lemmy problem. wondering if anyone has any experience or thoughts cuz its weird to have awful stuff hosted on one instance even though it was banworthy elsewhere.

edit 3: removed edit 2.

due to the offending user being banned from the federating instance before the post was removed? example

edit: i recognize this is not a local issue and may be more of a lemmy problem. wondering if anyone has any experience or thoughts cuz its weird to have awful stuff hosted on one instance even though it was banworthy elsewhere.

edit 3: removed edit 2.

Recently I was trying to reply to a comment and it wouldn't post the reply. Today I also wanted to upload a meme but the image wouldn't upload, so I resorted to using catbox.moe. Even with that, it still insisted on generating its own broken image and the thumbnail didn't generate either

UPDATE: Apparently only GIFs and replies are broken. Uploading a JPG or PNG works fine

https://lemmy.cafe/post/3646439

Recently I was trying to reply to a comment and it wouldn't post the reply. Today I also wanted to upload a meme but the image wouldn't upload, so I resorted to using catbox.moe. Even with that, it still insisted on generating its own broken image and the thumbnail didn't generate either

UPDATE: Apparently only GIFs and replies are broken. Uploading a JPG or PNG works fine

https://lemmy.cafe/post/3646439

It's the "silently" part that's the issue. I acknowledge that lemmy.cafe is entitled to defederate from whatever servers the administration pleases, but lemmy.ml still houses some of the largest communities in the Lemmyverse on some topics, and a heads-up that it was being blocked would have been appreciated.

It's the "silently" part that's the issue. I acknowledge that lemmy.cafe is entitled to defederate from whatever servers the administration pleases, but lemmy.ml still houses some of the largest communities in the Lemmyverse on some topics, and a heads-up that it was being blocked would have been appreciated.

The upgrade has gone through smoothly and everything seems to be running smoothly.

The performance looks to be better on the backend, time will tell if the memory leak issue is actually solved. So far, though - so good!