In reference to this thread:

• https://lemm.ee/post/44491669
• https://lemmy.world/post/20694710
• https://discuss.tchncs.de/post/23365372

Tl,dr:

• ClubsAll is a closed source platform which fetches Lemmy content
• ClubsAll however does allow Lemmy instances to federate its communities ([email protected] gives an error, see https://lemm.ee/post/44491669/15374449)
• Comments from ClubsAll do federate to Lemmy communities: (https://lemmy.world/comment/12828092 / https://clubsall.com/posts/what-food-and-or-drink-will-you-eat-when-you-are-senile-and-forget-you-already-had-it-the-meal-before-oD_s4)
• It is not possible to host a ClubsAll instance at the moment

What do you all think about this?

Been getting a lot of instability out of pictrs the last two days. Nothing in the week before that.

By instability, I'm talking 500 errors when uploading images (POST https://lemm.ee/pictrs/image). They seem pretty much random, I can get around the errors by just repeatedly trying to upload the same file until it works. As far as I can tell, there's nothing I'm doing wrong, trying to upload images too large produces a different, and much more reasonable, error. And there's no info at all in the response, except for the 500 code itself.

Without using external scripts, is there a way to hide posts that link to specific domains? For example, if I want to hide all posts where the OP put google.com links in the URL field when creating the post.

Following the request from https://lemm.ee/post/38756969 I want to add pronouns to my username. I would like have the full " [he/him]" string appended to my username, however lemm.ee only allows up to 20 characters.

Would it be possible to add more characters or a pronoun selector like on hexbear.net?

I currently have it blocked with ublock Origin, but I would like to be able to completely disable it so that post titles can use 100% of screen width.

cross-posted from: https://lemmy.dbzer0.com/post/25287498

cross-posted from: https://lemmy.zip/post/19638259

There are about 6 pages.dev domains spamming lemmy.world communities

The volume is definitely inorganic, and is across a wide range of communities

pages.dev is Cloudflare's site hosting which can be used for free - there are likely many legitimate sites that use that domain, but the current flood is suspicious

chronicleresolve.pages.dev

thefreedomproject.pages.dev

versarch.pages.dev

dailypulse.pages.dev

newssphere-6fu.pages.dev

iniko.pages.dev

miniza.pages.dev

orino.pages.dev

I'm cross posting because @[email protected] seems to be doing the same thing.

It might be an attack vector or something idk but better safe than sorry.

Not sure about this one but seems to be following same pattern.

@marvelous_[email protected]

~Anti~ ~Commercial-AI~ ~license~ ~(CC~ ~BY-NC-SA~ ~4.0)~

Hey folks!

Unfortunately, roughly 2 hours ago, lemm.ee went offline. The cause was our load balancer: it suddenly decided that all of our servers had become unhealthy, despite all health checks responding successfully when I requested them directly. In such cases, the load balancer stops serving all requests, effectively meaning that lemm.ee is unreachable for all users. I am still not sure what exactly caused the issue, but I will try to investigate more over the weekend.

For now, we have partially recovered, and I am continuing to work on remaining issues. Hopefully we will be back to 100% very soon. Sorry for the inconvenience!

It toggles between an open eye icon and and crossed out eye. What’s it do?

Edit: turns out it hides hidden posts.

Is anyone else experiencing slow refreshes and even slow up/down-votes on this instance? It does not seem to be constant, but I've noticed this for the past couple of days.

After being helped several times with my various community issues by our kind site-runner, I would like to make a useful donation at this time. Oy, but there seems to be a problem:

Now, I use an add-on in Chrome called "uMatrix," which is a script-blocker. This add-on forces me to hand-enable whatever necessary java-scripts there might be on sites, but it's not perfect. Okay, fine, so in Ko-fi's case, I wound up having to turn the tool completely OFF in order to get to the final payment-step, for anyone reading who had issues with such.

Now, to the final step:
My donation / pay options are evidently these: iDEAL, Bancontact, Przelewy24 and EPS.

New problem: I have utterly no idea what those are.
Is there a way then to donate via PayPal, perhaps?

EDIT: Sorry, I guess this should have gone in "Support." Please do move as necessary.

Hexbear disappeared for a while. I guess they're federating with us again? What changed to make them want to do that again?

I always get the “file size too large” or whatever the error is when trying to post images as comments. I’ve asked and was told that it’s lemm.ee’s limitation on posts/comments.

Hey folks

Just a heads up that I will be doing some minor database maintenance shortly. I expect the downtime to last <5 minutes.

Have a nice day!

Update: maintenance is complete!

When I select the cross post option everything looks as normal. When I select the community I want to cross post to, all information in fields above drop-down is deleted. Tried copying the URL and re-pasting in the url field but the system refuses to generate the title after selecting the "generate title" option that is offered.

I am operating on a phone via mobile site so my workaround options are limited

Link previews seem to be broken since upgrade. As you can see this isn't from all instances so is this a federation issue or an upgrade issue?

For what it's worth I get the same experience in Voyager app on Android as well as web-ui.

Anyone else experiencing the same?

Hey there! I recently got server banned b/c I was sharing a link in men's communities and I was mistaken for a bot.

I didn't have a way to respond to the ban. I was just logged out. I didn't see a link or even which mod did the ban so that I could reply to them, but then I couldn't get into my account to reply anyway.

It'd be nice to maybe get a "if you think this was in error" email or something.

Although I know that if lemmy was the size of reddit, people would hook that email up to chatgpt, so :/

Anyway, it'd be a nice feature to have. I doubt I'll need it again, but I was quite confused as to what to do.

Hey all!

Upcoming lemm.ee cakeday

Can you believe that lemm.ee is almost 1 year old? In just a couple of weeks (specifically, on the 9th of June), we will be able to celebrate our first instance cakeday.

I am thinking of compiling some stats about how lemm.ee has been used in its first year, if you have any specific stats in particular you would like to see, feel free to comment below. I will try to accommodate any ideas as I start gathering this info!

Infrastructure updates

A few weeks ago, I posted about plans to make some changes to our infrastructure in order to deal with different intermittent networking issues.. It took a bit longer than I hoped (just did not manage to get enough free time between then and now), but I am happy to report that this work has now been completed! Additionally, I have decommissioned our stand-alone pict-rs server.

With the two changes mentioned above, I believe lemm.ee should now be much more resilient going forwad, and I expect a significantly lower rate of infrastructure-related issues for the rest of the year!

I'll leave a tehcnical overview about the problem & solution below for those interested, but if these details don't interest you, then you can safely skip the rest of this post.

For context, lemm.ee has been hosted on Hetzner servers for most of this year (having migrated from DigitalOcean initially), with everything except our database being hosted on the Hetzner Cloud side, and the database itself living on a powerful dedicated Hetzner server. This mix allows a great amount of flexibility for redeploying and horizontally scaling our application servers, while still allowing a really cost-effective way of hosting a quite resource-hungry database.

In order to facilitate networking between the cloud servers and the dedicated database server (which live in different networks), Hetzner provides a service named "vSwitch". This service basically allows you to connect different servers together in a private network. Unfortunately, I discovered quite quickly that this service is very unreliable. During the short few months that we have been using the vSwitch, we have gone through one extended period of downtime (where the service was just completely broken for several hours), as well as dozens (if not hundreds at this point) intermittent disconnects, where servers randomly lose their connections over the vSwitch. After such a disconnect, the connection never recovers without manual intervetion.

For most lemm.ee users, the majority of these vSwitch issues have been mostly invisible, as we have redundancy in our servers - if one server loses its connection to the database, other servers will take over the load. Additionally, I have generally been able to respond quite quickly to issues by redeploying the broken servers (or deploying other temporary workarounds). However, in addition to a huge amount of these issues which lemm.ee users hopefully haven't ever noticed, there have also been a few short periods of downtime this year so far, as well as a few cases of federation delays. These more extreme cases were generally caused by multiple servers losing their vSwitch connections at the same time.

After several attempts to work around these issues, I decided that we need to migrate away from vSwitch.

As of earlier today, lemm.ee is no longer using Hetzner's vSwitch at all!

I finally found enough time earlier today to focus on this migration, and I was able to successfully complete it. None of our networking is relying on the vSwitch anymore.

In the end, I went with quite a simple solution - I configured a host-level firewall (nftables) on our database dedicated server, which will deny all connections by default. Whenever any cloud servers are added/removed, their corresponding public IP addresses are added/removed in the allowlist of our database firewall. It would have been ideal to do this whole logic in Hetzner's own firewall, but that one unfortunately has a limit of only 10 rules per server, which is just not enough for our setup.

Bonus: our pict-rs server has been decommissioned!

Pict-rs is the software which Lemmy uses for everything related to media (image storage mostly). Initially, pict-rs required a local filesystem to store both files as well as metadata about files. Since the beginning, lemm.ee has used a dedicated server just for pict-rs, in order to ensure we could easily redeploy the rest of our servers without losing any images.

Over the past year, pict-rs has gained the ability to store files in object storage, and metadata in a PostgreSQL database. This meant that the server running pict-rs itself no longer contained any of the important data, so it became possible to redeploy without losing any images. Additionally, this meant that it would be possible to run multiple pict-rs servers in parallel.

While we had already migrated our pict-rs server to use object storage and PostgreSQL several months ago, we still had the single dedicated pict-rs server up until today. I have been planning for a while to decommission this server, and start running pict-rs directly on each one of our Lemmy application servers. Earlier today, I was able to complete this plan. This should hopefully mean that our pict-rs server is less likely to get overloaded, and it also means a tiny reduction in our overall monthly infrastructure bill (due to one less server running).

With the above changes, I think our infrastructure has become more robust, and hopefully, we will experience less issues with images, federation, and general downtime going forward.

That's all from me for now. Feel free to leave any thoughts or questions in the comments, and as always, I hope you're having a great day!

Hey folks!

This is a quick notice about a change to our moderation policy.

We have had a policy on lemm.ee for administration and federation nearly since the very beginning. This policy has also always included a section about moderator responsibilities. Today, we have made two changes to this policy:

1. The policy has been renamed to Policy for administration, moderation, federation - this is to make it clear that the policy is also relevant for mods
2. We have introduced a new responsibility for moderators, they must "Ensure that they only provide accurate and clear reasons for mod actions".

The reason for the addition is that mod log actions federate out to other instances, and are more or less permanent (due to how Lemmy and federation works right now). This means that users do not really currently have any easy way to clarify or defend themselves against inaccurate accusations in the mod log.

As always, I am very grateful to all mods for your efforts in building awesome communities on lemm.ee. I hope you can understand why this new policy is necessary - I do not want to make your lives more difficult, the goal is to just try and reduce any mod log related misunderstandings in the future.

Thank you for reading and have a nice day!

My community is:
https://lemm.ee/c/eurographicnovels

The post I was working on was:
https://lemm.ee/post/2890991

To be clear-- I, in no, way, shape or form intend to delete my community. I wish the community to remain undeleted, thanks.

In case it matters:

57 users / day
150 users / week
444 users / month
1.53K users / 6 months
635 subscribers
317 Posts
902 Comments

EDIT: Google retains the specific URL's of a bunch of our posts, such as the "Moebius" ones.

Hey folks!

We unfortunately had about half an hour of unplanned downtime today. This was caused by an issue with our hosting provider. The issue is solved for now, and I am planning to make some changes to prevent similar issues in the future. Sorry for the inconvenience!

Technical details

Our servers are communicating with our database over Hetzner's "vSwitch" service. Unfortunately, this service seems to be quite flaky - over the past few months, I have had to deal with the connection just dropping without recovering many times. Mostly this has not resulted in any noticeable downtime, as we have redundant servers, so even if one of them stops working, it won't affect lemm.ee users. However, in this instance, all of our API servers lost their connection to our database at the same time, which resulted in actual downtime.

I have now decided to migrate our setup away from the vSwitch in the near future to hopefully stop these issues for good. Should be possible to do this migration without any downtime, I just need to set aside some time to actually create an alternative solution for us, most likely over the coming weekend. I will update this post once the migration is complete.

Update: the migration is now complete! You can read more here.

EDIT: I should have posted this in the Support community - as others are also doing.

https://lemm.ee/c/support

Just noticed tonight that when I visit my FullMoviesOnYouTube community, the banner image is not shown.

I tried linking directly to it:

https://lemm.ee/pictrs/image/642b333b-5c37-4d39-af9f-cc876de484fc.webp?format=webp

And got this error:

{"error":"unknown","message":"Request error: error sending request for url (http://10.0.0.3:8080/image/process.webp?src=642b333b-5c37-4d39-af9f-cc876de484fc.webp): operation timed out"}

Any idea what's up? It's definitely been working as of just last week.

Hello!

I noticed that a post I created from my sopuli account to a lemm.ee community, federated over here, but then not out any other instances.

Is lemm.ee dealing with similar federation woes as lemmy.world?

I was using imgur now to upload pictures, but for none of my posts it shows a thumbnail for me in jerboa. It does at least for some of those using lemmy.world, so i assume this may be something lemm.ee specific? Thumbnails seem to be shown for images hosted on catbox.moe or other instances. I just don't get what the difference is.

What is it that would make thumbnails show up?

Hey folks

This is just a quick heads up that I need to perform some maintenance & upgrades on our database server, which unfortunately will require downtime. I don't expect the downtime to last for longer than 2-3 minutes, but just wanted to give a heads up first so you know not to be concerned.

That's all, hope you have a great week!

Edit: maintenance complete!

There's presently [email protected] and [email protected], but both have low activity and appear to be unmoderated. Removing communities is probably reserved as a last resort, and seeking some moderators may be preferable, so I thought I might ask about'em to raise some attention.

These are the kinds of communities that could get pretty rough if left in their current unattended state, I think.