Programming.dev Meta

Yeah, I actually think lemmy could benefit from improved scraping and indexing. For example, it'd really help if more search engines could natively understand Lemmy's federated nature, e.g:

• Deduplicate links by prioritising results for instances hosting the community that a post was originally submitted to.
• Include and denote cross posts by recognizing order of submission timestamp and prioritizing popularity via vote ratios, comment counts, and lurker click-through traffic.
• Do the same deduplication and prioritization across instances, but for comments as well.

Another use case besides search engines would be for internet archive projects, helping to preserve historic internet content even in the face of lemmy instances falling offline and disappearing. For example, much knowledge was lost to us due to the Twitter APIoplicips and Reddit Blackout: E.g:

• https://wiki.archiveteam.org/index.php/Warrior_projects
• https://tracker.archiveteam.org/reddit/

Most of the above will only ever be possible due to improved scraping or even federation APIs.