Meta

623 readers
1 users here now

Discussion about the aussie.zone instance itself

founded 1 year ago
MODERATORS
1
2
submitted 1 year ago* (last edited 1 week ago) by [email protected] to c/[email protected]
 
 

Changelog

2024.10.25 - migrated pict-rs image store to its built in object storage
2024.10.24 - banned lemmit.online instance, only used to spam content from reddit into Lemmy. If you want that content, use reddit directly.
2024.10.23 - DB config changes, spun up worker instances of lemmy service, re-enabled "ProxyAllImages"
2024.10.12 - disabled the "ProxyAllImages" configuration to try and resolve the long running image issues
2024.06.26 - upgraded lemmy to 0.19.5
2024.06.09 - upgraded lemmy to 0.19.4, postgres to 16, pict-rs to 0.5.15
2024.01.23 - upgraded lemmy to 0.19.3
2024.01.11 - upgraded lemmy to 0.19.2
2023.12.21 - upgraded lemmy to 0.19.1
2023.12.17 - upgraded lemmy to 0.19.0
2023.10.12 - upgrade VPS to 160GB, other specs unchanged
2023.08.9 - upgraded lemmy and lemmy-ui to 0.18.4
2023.07.29 - upgraded lemmy and lemmy-ui to 0.18.3
2023.07.11 - upgraded lemmy and lemmy-ui to 0.18.2
2023.07.10 - upgraded lemmy-ui to 0.18.2-rc.1 to mitigate XSS vulnerability
2023.07.10 - VPS upgraded to 8GB RAM (required to upgrade storage, needed anyway.. only $2pm)
2023.07.08 - upgraded lemmy and lemmy-ui 0.18.1 ๐ŸŽ‰
2023.07.06 - upgraded lemmy to 0.18.1-rc.10 and lemmy-ui 0.18.1-rc.11
2023.07.04 - upgraded lemmy-ui to 0.18.1-rc.10
2023.07.04 - upgraded lemmy and lemmy-ui to 0.18.1-rc.9
2023.07.03 - upgraded Lemmy to 0.18.1-rc.4 and lemmy-ui 0.18-rc.7
2023.06.28 - VPS upgraded from 2 to 4 vCPU, other specs the same.
2023.06.24 - upgraded Lemmy to 0.18.0
2023.06.23 - upgraded Lemmy to 0.18.0-rc.6
2023.06.13 - upgraded Lemmy to 0.17.4. Increased federation workers and reduced logging storage at the same time.
2023.06.10 - VPS storage upgraded from 40GB to 80GB, other specs the same.
2023.06.09 - VPS upgraded to 4GB RAM, 2 vCPU, other specs the same.
2023.06.08 - aussie.zone created, running Lemmy 0.17.3. OVH VPS was 2GB RAM, 1 vCPU, 40GB NVME

Nerd Stuff

The aussie.zone server is currently an OVH VPS in Sydney:
8GB RAM, 4 vCPU, 160GB NVME storage

Images are stored in an object store bucket on Wasabi, also in Sydney.

I post updates ~~every week or so~~ randomly with current server resource graphs:
Nerd update 20/4/24
Nerd update 2/9/23
Nerd update 13/8/23
Nerd update 5/8/23
Nerd update 29/7/23
Nerd update 22/7/23
Nerd update 15/7/23
Nerd update 7/7/23
Nerd update 30/6/23

If you have any questions, please post.

2
10
submitted 2 days ago* (last edited 1 day ago) by [email protected] to c/[email protected]
 
 

I've spun this up for fun, to see how it compares to the base lemmy UI. Give it a whirl, and post any feedback in this thread. Enjoy!

It could go down at any time, as it looks as though the dev is no longer maintining it...

edit: using this https://github.com/rystaf/mlmym

3
 
 

Its been 6 months or so... figure its time for another of these. Keep in mind there have been some major config changes in the last week, which has resulted in the oddities below.

Graphs below cover 2 months, except Cloudflare which only goes to 30 days on free accounts.

CPU:

Memory:

Network:

Storage:

Cloudflare caching:

Comments: The server is still happily chugging along. Looking even happier now that I've properly migrated pict-rs to its integrated object storage config, rather than the bodged up setup.

RAM/CPU are all fine. Storage use is growing slowly as various databases grow. Still a long way from needing to purge old posts, if ever.

Cloudflare is saving less traffic these days, since Lemmy added support to proxy all images. Not a concern, well under the bandwidth cap for the server.

As usual feel free to ask any questions.

4
37
Pictures are broken (aussie.zone)
submitted 1 week ago* (last edited 1 week ago) by [email protected] to c/[email protected]
 
 

I'm in the process of migrating images to a properly configured object storage setup. This involves an offline migration of files. Once complete, I'll start up pict-rs again. Until then, most images will be broken.

All going well this will finish by morning Perth time, and once up and running again may help with the ongoing issues we've had with images.

5
 
 

Over the last few days I've noticed that I'm semi-regularly getting Cloudfare error pages with timeout errors when I go to load a page here. Once the site loads it normally works fine for a while. Sounds similar to Baku's post from a few months ago, if others are experiencing the same at the moment possibly a reboot would help here too?

6
 
 

Leading on from an issue with some lost stickied posts on [email protected] , i'm now gathering together useful sources of information in regards the Aus environment.

On !perth i gathered them in the sidebar, so they should remain if i decide one day to remove my account.

I'm currently thinking to do the same thing with [email protected] since i've lost those good stickied posts anyway.

My question, probably for [email protected] or [email protected] , is, is there an ability to add drop down menus to the sidebar to enable better organisation of the links.

If you look at the !perth sidebar its already a bit messy and unwieldy, i assume [email protected] will only be more so, with the amount of source links likely to be added.

Edit: sorry for all the stuffed links, i'll try to fix them

7
 
 

@[email protected]'s comments were deleted in a thread I was just looking at, and I went onto their profile and it also appears to have been deleted? Anyone have any knowledge about them? It's sad if we've lost such a long-time excellent member of this community.

8
 
 

After some users have had issues recently, I've finally gotten around to putting in place a better solution for outbound email from this instance. It now sends out via Amazon SES, rather than directly from our OVH VPS.

The result is emails should actually get to more people now, rather than being blocked by over-enthusiastic spam filters... looking at you Outlook and Gmail.

9
 
 

I've noticed a lot of broken images around the place and can't quite work out what's happening. It doesn't seem to be a caching issue, at least not on my end, as trying to visit the image URL directly returns a 404.

As noted in a post in the Melbourne daily thread a few weeks ago, there are also a lot of broken community icons about the place. And as I think was raised a while ago, thumbnails seem to be broken most of the time as well.

As an example, here's what the trending communities tab looks like:

The communities page is a little better, but it's about 50/50 as to whether the icon does or doesn't load:

Here's a list of my own posts with missing images I've noticed recently

Post Image
https://aussie.zone/post/13071169 https://aussie.zone/pictrs/image/2982b6d9-b84a-4c2f-8f3b-1f38ecd27241.png
https://aussie.zone/post/12388426 https://aussie.zone/pictrs/image/8638242e-4d62-4372-9a7f-15c913fc7b83.jpeg
https://aussie.zone/post/11404847 https://aussie.zone/pictrs/image/d22d1f8f-e288-4190-8df5-ef4e6756c396.jpeg
https://aussie.zone/post/11369832 https://aussie.zone/pictrs/image/a9d77a68-3a9b-4f4d-9508-fc6aefa14434.jpeg
https://aussie.zone/post/11285649) https://aussie.zone/pictrs/image/8a64f759-77e6-4e38-a739-2f875e90cbf0.png
https://aussie.zone/post/10604574 https://aussie.zone/pictrs/image/39091834-d6fd-45c1-9520-73b9b7be3529.jpeg
https://aussie.zone/post/10981887 https://aussie.zone/pictrs/image/b4df332b-82a9-4ec8-b115-ac1b1b1824b0.jpeg

That last image seems to be super broken at the time of writing. Instead of just returning a generic 404, it returns:
{"error":"unknown","message":"Request error: error sending request for url (http://pictrs:8080/image/original/b4df332b-82a9-4ec8-b115-ac1b1b1824b0.jpeg): connection error: Connection reset by peer (os error 104)"}

Additionally, this is how thumbnails on links have been previewing:

Right-clicking and opening in new tab, it seems to be pulling a random pixel from somewhere in the image and then enlarging it and proxying it (500% zoom):

Example URL

But then other URL thumbnails don't work at all, for instance this post on trending:

Which links to Wikipedia, but fails for whatever reason, then tries to load an image called "broken-image-fallback.png", and, ironically, can't even load that (leads to a 404 page).

And some thumbnails on working images don't work, for example this post in c/GardeningAustralia doesn't load - even if I click on the thumbnail (which should normally load the full size/original image), but does work if I click the image URL itself.

One more thing: this new image proxy thing is getting super annoying. I realise it's a Lemmy-side implementation issue, but a couple of sites (Imgur, for example), don't like all the image requests from a single server/IP and end up rate limiting us. This effectively means that Imgur is unusable and no images load.

I actually thought there might have been another mass image purge like last years' incident which required the purging of 48 hours of images, but there hasn't been any announcements about it, and the other broken images dotted around the joint makes me think Pictrs is on its deathbed.

Any ideas what's going on?

Sidenote: image uploads whether in posts or as embedded images within posts/comments seem to fail about half the time for me on both mobile and desktop. I thought this was an issue with Boost again, but it's happening to me on PC as well. On mobile, it usually either returns no error at all, or a 500. On PC, no UI errors are displayed, it just doesn't embed the image, but network tools tell me that something somewhere is returning a 500, likely the image uploader thingy.

10
 
 

I thought the recent patches were supposed to fix the slow federation, but we're still not getting comments or vote tallies through for over a week.

11
6
URL question. (aussie.zone)
submitted 1 month ago* (last edited 1 month ago) by [email protected] to c/[email protected]
 
 

My web dunce question of the week is,

Why would the images url be linked to aussie.zone when the community is blahaj.zone and the poster is lemm.ee?

Is it as simple as the lemm.ee poster might have seen it on aussie.zone and cross posted, or copied over from there?

Edit,

So, just checked this post on the sync app, (using Jerboa for the post), and sync has recognised the image as a URL? I'm confused, i loaded it in Jerboa like i'd normally load a screenshot as an image. Anyway, sorry if the picture doesn't display.

I'm just trying to understand the lemmy system a little better, not trying to highlight a perceived problem.

12
11
REBOOTING (aussie.zone)
submitted 2 months ago by [email protected] to c/[email protected]
 
 

About to reboot the server, hold onto your hats.

13
 
 

I've been seeing various 5xx error codes (usually 504s, but occasionally 502) sporadically most of the day. It doesn't seem to be a constant thing, just every 5-10 or so loads/clicks, it errors out. I've also noticed it's sometimes taking a while to load. Seems 50/50 whether it takes 1-2 seconds, or 6-10 seconds to load.

Is this something on my end, or is today a lazy Sunday for the server, too?

(I took a screenshot, but I'm getting errors when I try to upload that, too. A classic "ffprobe failed with exit code 1: success")

14
 
 

I'm the developer of Fediverser Project, which is a set of services to make it easy for people on Reddit to migrate to the Fediverse. It lets people use their Reddit credentials (OAuth) to sign up and create an account on a Lemmy server.

It also offers a cool onboarding feature: during signup, we can fetch the user's subscribed subreddits, and we use this information to automatically subscribe them to the corresponding Lemmy (or Kbin/Mbin) community. This "subreddit -> fediverse group" map is crowdsourced and people can sign up if they want to contribute. The "main" site also provides a "Find an instance" feature: it can track all the servers that use the Fediverse software and redirect users to their closest instance.

To enable this service, the Lemmy admin needs to add a couple of docker services to their setup and needs to get their own Reddit API key (which is used only for authentication, so well within the rate limits and certainly not incurring any prices).

I'd really like to see aussie.zone becoming part of the network. I believe this would make it faster and simpler to get more people in the fediverse, and I'm willing to provide all the support and help needed to get the "country-based" services getting started with it.

Any questions, don't hesitate to ask.

15
 
 

This is a simple software bug. You see, the way Lemmy federates is that updates to a community only federate to instances that have a user subscribed to that community. If you take a community like [email protected], I'm the only person on all of Aussie Zone who subscribed to it. I know it because when I subscribed, we started getting updates from Blahaj.Zone about that community.

A few days ago, I was mistakenly banned from that community. And this caused Blahaj.Zone to stop sending our instance updates on the community. I spoke with the admin of the community and got unbanned, but the unban didn't federate. The instance will only let us know I'm unbanned if someone here subscribes over there. And I can't subscribe, because this server thinks I'm banned. It's a deadlock. If it wasn't for the ban, I could subscribe and the server would realise there isn't a ban anymore. But I can't, because this server thinks there is still a ban.

16
 
 

When I try and upload an image for my avatar, I receive the message:

{"data":{"files":null,"msg":"ffprobe Failed with exit status: 1"},"state":"success"}

This happened with both a WebP and a PNG file, so I suspect it's a bug.

I can upload images fine here, and when I export my account details it shows a valid image URL for my avatar.

17
 
 

So, as per images below, when you search for an Australian community associated with lemmy, lemmy.world is more likely to come up than Aussie Zone in all i've tried, bar Melbourne our most active community.

My question: Is this a problem we should consider intentional action to correct? And if so what could we do?

18
 
 

Sorry if this has been asked/answered previously, but it just struck me that I hadn't really noticed any impacts of CloudStrike on aussie.zone.

Was wondering if it was a non-event, or maybe there are some war stories with heroes deserving medals!

19
 
 

So, I've just done something that I said I'd be reluctant to do in future without community consultation and blocked an instance.

I don't think anyone will have any issues with it, but I figured I'd let everyone know since this executive decision affects all aussie.zone users.

Hate content

20
 
 

Would people be interested in a community dedicated to discussing lemmy's in general but, specifically aussie zone's growth?

I'm thinking a pretty wide ranging set of ideas including,

  • exposure and promotion of fediverse-lemmy-aussie zone,
  • discussions on server structures and dispersal of user-bases,
  • philosophical discussions on growth as an objective in the case of lemmy,
  • discussions on different platforms aussie zone users might want to gather round as the potential use cases of the federated social web start to be explored.

I'm thinking like [email protected] , but a sister community that has regard to aussie zone user's corner of the fediverse.

To be used by aussie zone as a general compass setting and project brainstorming community for how we might like to develop, if at all.

Or is [email protected] the better place for these sorts of discussions?

21
 
 

There's no communities I think need better moderation, or any that have no active mods (that I know of), but if there is one day, can somebody make a request to have ownership or moderation powers transferred over to them?

22
 
 

Hey all, following the work over the weekend we're now running Lemmy 0.19.4. please post any comments, questions, feedback or issues in this thread.

One of the major features added has been the ability to proxy third party images, which I've enabled. I'll be keeping a closer eye on our server utilisation to see how this goes...

23
71
Maintenance (aussie.zone)
submitted 4 months ago* (last edited 4 months ago) by [email protected] to c/[email protected]
 
 

This weekend I'll be working to upgrade AZ to lemmy 0.19.4, which requires changes to some other back end supporting systems.

Expect occasional errors/slowdowns, broken images etc.

Once complete, I'll be making further changes to enable/tweak some of the new features.

UPDATE: one of the back end component upgrades requires dumping and reimporting the entire lemmy database. This will require ~1 hour of total downtime for the site. I expect this to kick off tonight ~9pm Perth time.

UPDATE2: DB dump/re-import going to happen ~6pm Perth time, ie about 10 minutes from this edit.

UPDATGE3: we're back after the postgres upgrade. Next will be a brief outage for the lemmy upgrade itself... after I've had dinner ๐Ÿ™‚

UPDATE34: We're on lemmy 0.19.4 now. I'll be looking at new features/settings and playing around with them.

24
 
 

I know there's been a few posts about this before, but it's been a month since the last one and it's still ongoing. It doesn't seem that any of the LW admins responded to Zag's post on their help community, and the last response from lodion/Nath I'm aware of was from 4 months ago when there were outright federation failures as opposed to just lengthy delays.

@[email protected] posted a comment on the post from last month about the delays stating that it's an issue on our end as our server isn't keeping up. I'm not sure whether this is the case or not, and I'm not sure how to interpret the Grafana dashboard they linked to, but as it's a new reply on an old post, I wanted to note it.

Current federation delays seem to be around 7 days. It doesn't seem to be affecting posts themselves on Lemmy.world communities, but does affect all replies to them (even from users on other instances), and all upvotes on the posts. [Edit: on further investigation, this isn't the case. The current delays are at least 13 days, and this does actually affect posts too]

I don't want to sound too pushy, since the LW admins and Lodion/Nath are all volunteers, but I was hoping we might be able to get an update on what the cause is, and if it's an issue in Lemmy itself, if anybody's opened an issue on GitHub and the developers are aware.

(NB: I don't interact that much with LW, so all of my testing has been on the Boost for Lemmy community.)

25
 
 

Hey all, can anyone tell me the date we're gona land on for the first anniverssary?

I guess it'll be when the instance was spun up? Or i don't know, maybe when it was a twinkle in Lodion's eye?

view more: next โ€บ