Lemmy.world Support

3202 readers

2 users here now

Lemmy.world Support

Welcome to the official Lemmy.world Support community! Post your issues or questions about Lemmy.world here.

This community is for issues related to the Lemmy World instance only. For Lemmy software requests or bug reports, please go to the Lemmy github page.

This community is subject to the rules defined here for lemmy.world.

To open a support ticket

You can also DM https://lemmy.world/u/lwreport or email [email protected] (PGP Supported) if you need to reach our directly to the admin team.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

founded 1 year ago

MODERATORS

[email protected]

What is the rate limit for requests on the webrequests for search and API for lemmy.world? (lemmy.world)

submitted 1 year ago by [email protected] to c/[email protected]

3 comments fedilink hide all child comments

I would like to try building a search index for this instance (maybe others) and as such would like to crawl the site with automated spiders. Now with the shutdown of the reddit API I expect the site to come under quite substantially load and also I would ofc try to not spam the site with too many requests as to not get banned or blocked, due to looking like a DOS attack. Could anyone provide some information on this?

top 3 comments

sorted by: hot top controversial new old

[–] [email protected] 0 points 1 year ago (1 children)

Many existing Fediverse services are being operated by people who are opposed indexing the content on their instance(s). You may run into resistance from that angle.

[–] [email protected] 1 points 1 year ago (1 children)

I mean unless they make their instance private I don't see why you wouldn't index them? That's literally why google provided such a value in their early days.

[–] [email protected] 1 points 1 year ago

Even Google doesn't index webpages that include "noindex" in a header. You are going to run into a lot of people who don't agree with what you are trying to do. If you start reaching out to the people running Fediverse services to let them know that you're trying to index the data on their services, you can learn what they think of the idea.