this post was submitted on 05 Oct 2023

321 points (98.5% liked)

Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ

53939 readers

296 users here now

⚓ Dedicated to the discussion of digital piracy, including ethical problems and legal advancements.

Rules • Full Version

1. Posts must be related to the discussion of digital piracy

2. Don't request invites, trade, sell, or self-promote

3. Don't request or link to specific pirated titles, including DMs

4. Don't submit low-quality posts, be entitled, or harass others

Loot, Pillage, & Plunder

💰 Please help cover server costs.

founded 1 year ago

MODERATORS

[email protected]

321

Introducing Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration (bitmagnet.io)

submitted 11 months ago by [email protected] to c/[email protected]

19 comments fedilink hide all child comments

cross-posted from: https://lemmy.world/post/6301281

I'm excited to announce the first alpha preview of this project that I've been working on for the past 4 months. I'm initially posting about this in a few small communities, and hoping to get some input from early adopters and beta testers.

What is a DHT crawler?

The DHT crawler is Bitmagnet’s killer feature that (currently) makes it unique. Well, almost unique, read on…

So what is it? You might be aware that you can enable DHT in your BitTorrent client, and that this allows you find peers who are announcing a torrent’s hash to a Distributed Hash Table (DHT), rather than to a centralized tracker. DHT’s lesser known feature is that it allows you to crawl the info hashes it knows about. This is how Bitmagnet’s DHT crawler works works - it crawls the DHT network, requesting metadata about each info hash it discovers. It then further enriches this metadata by attempting to classify it and associate it with known pieces of content, such as movies and TV shows. It then allows you to search everything it has indexed.

This means that Bitmagnet is not reliant on any external trackers or torrent indexers. It’s a self-contained, self-hosted torrent indexer, connected via the DHT to a global network of peers and constantly discovering new content.

The DHT crawler is not quite unique to Bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature. However that project is no longer maintained, and does not provide the other features such as content classification, and integration with other software in the ecosystem, that greatly improve usability.

Currently implemented features of Bitmagnet:

A DHT crawler

A generic BitTorrent indexer: Bitmagnet can index torrents from any source, not only the DHT network - currently this is only possible via the /import endpoint; more user-friendly methods are in the pipeline

A content classifier that can currently identify movie and television content, along with key related attributes such as language, resolution, source (BluRay, webrip etc.) and enriches this with data from The Movie Database

An import facility for ingesting torrents from any source, for example the RARBG backup

A torrent search engine

A GraphQL API: currently this provides a single search query; there is also an embedded GraphQL playground at /graphql

A web user interface implemented in Angular: currently this is a simple single-page application providing a user interface for search queries via the GraphQL API

A Torznab-compatible endpoint for integration with the Serverr stack

Interested?

If this project interests you then I'd really appreciate your input:

How did you get along with following the documentation and installation instructions? Were there any pain points?

There's a roadmap of high-priority features on the website - what do you see as the highest priority for near-term development?

If you're a developer, are you interested in contributing to the project?

Thanks for your attention. If you're interested in this project and would like to help it gain momentum then please give it a star on GitHub, and expect further updates soon!

all 20 comments

sorted by: hot top controversial new old

[–] [email protected] 22 points 11 months ago (1 children)

This looks rad. Sadly I don't have time to pay around with it right now but will def come back to this.

[–] [email protected] 24 points 11 months ago

It’s been 2 hours, come back to this.

[–] [email protected] 13 points 11 months ago (4 children)

Slight off-topic but do people not know about Tixati ? It's a torrent client but it has searchable DHT (by torrent name) and full decentralised forums (with audio streaming, for whatever reason lol) for a long time now.
Except for being closed source it's pretty interesting. A bit baffled there isn't more talk about it, or at least about the features.

[–] [email protected] 17 points 11 months ago

As a people; no, I did not.

[–] [email protected] 6 points 11 months ago (1 children)

Great client but unfortunately for those of us on private trackers it isn't going to be whitelisted on many of them, or at least the most prominent ones. At least that was the case last I checked, and I gave up on it after.

[–] AdmiralShat 2 points 11 months ago (1 children)

What's the actual benefit for private trackers? I've never failed to find a torrent for something I wanted, so I don't know what they actually bring to the table

[–] [email protected] 3 points 11 months ago (1 children)

Always high quality content and easy as hell to find. Flacs for music, Blu-ray sourced movies/tv with high bitrate, etc. No sifting through torrents of questionable quality.
That's because good private trackers have torrent checkers, report systems, and staff/dedicated users making sure content is up to par in terms of quality and safety against malware.
They are filled with seed boxing users making all of your downloads very fast. On the flip side these users often cannabalize upload ratio for torrents so individuals on normal connections might have to rely on systems outside purely seeding to maintain a proper ratio. (This is exaggerated on places like r/trackers though, it's not hard at all imo.)
The request sections can get you pretty much anything that's reasonably obtainable if you out a bounty up. For me, I have niche tastes in music so that's been helpful more than a few times.
I never have to worry about my isp complaining about what I download. I've never received a complaint once, and I've never used a VPN.

[–] AdmiralShat 1 points 11 months ago* (last edited 11 months ago) (1 children)

Thanks for the response

On that last point, is that still a thing? I haven't gotten a torrent complaint in over a decade now, I honestly that was like a Comcast only thing at this point.

[–] [email protected] 2 points 11 months ago (1 children)

Comcast is the only option where I live. And yes, there are more strict rules in certain other regions like I think some European countries. Many people I've talked to have gotten isp letters just where I live though.

[–] AdmiralShat 1 points 11 months ago

Does your power company have any plans for internet? Mine installed fiber and I know many across the US have also done the same.

[–] [email protected] 5 points 11 months ago (1 children)

Until I was reading about this project I had never heard of it. And I would consider myself pretty plugged into torrent news.

[–] [email protected] 2 points 11 months ago

Proprietary?

[–] [email protected] 8 points 11 months ago

This is dope! There are a lot more important things on the roadmap, but a qBitTorrent plugin or compatible endpoint would be amazing.

[–] [email protected] 7 points 11 months ago

The included docker compose was very easy to use. I was up and running in just a few minutes.

DHT crawling started immediately which was pretty cool to see.

Sometime later this week I'll try integrating it into my arr stack.

[–] [email protected] 5 points 11 months ago

Is it safe to run this without a VPN if I am just using it to index?

[–] [email protected] 5 points 11 months ago* (last edited 11 months ago)

What kind of storage usage can I expect from running this? I would also assume that the database would heavily prefer solid state storage?

Edit: the author answered that question here already: https://lemmy.world/comment/4148270

[–] [email protected] 4 points 11 months ago* (last edited 11 months ago)

Very nice work, and thanks for sharing it!

I haven't spent much time digging through the code yet but noted that you're running your application in the ENTRYPOINT rather than the CMD. This can make overriding or extending parts of your container difficult, so you may wanna change that.

I may try setting this on my k8s cluster... if it can build on aarm64 ;-)

[–] [email protected] 3 points 11 months ago

Please, with sugar on top: make this a plugin that can be integrated to qbittorrent or rtorrent.

Try to make your idea reach the seedboxes. This would be great for open-tracker seedboxes!

[–] [email protected] 1 points 11 months ago

Looks like a great replacement for rats on the boat.