this post was submitted on 29 Jan 2025

757 points (97.0% liked)

Technology

61774 readers

3499 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each other!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed
Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago

MODERATORS

[email protected]

757

AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt (arstechnica.com)

submitted 1 week ago by [email protected] to c/[email protected]

86 comments fedilink hide all child comments

Building on an anti-spam cybersecurity tactic known as tarpitting, he created Nepenthes, malicious software named after a carnivorous plant that will "eat just about anything that finds its way inside."

Aaron clearly warns users that Nepenthes is aggressive malware. It's not to be deployed by site owners uncomfortable with trapping AI crawlers and sending them down an "infinite maze" of static files with no exit links, where they "get stuck" and "thrash around" for months, he tells users. Once trapped, the crawlers can be fed gibberish data, aka Markov babble, which is designed to poison AI models. That's likely an appealing bonus feature for any site owners who, like Aaron, are fed up with paying for AI scraping and just want to watch AI burn.

top 50 comments

sorted by: hot top controversial new old

[–] [email protected] 10 points 6 days ago

manual and builds are here: https://zadzmo.org/code/nepenthes/

[+] [email protected] -39 points 1 week ago (5 children)

So instead of the AI wasting your resources and money by ignoring your robots.txt, you're going to waste your own resources and money by inviting them to increase their load on your server, but make it permanent and nonstop. Brilliant. Hey, even better, you should host your site on something that charges you based on usage, that'll really show the AI makers who is boss. 🤣

[–] [email protected] 33 points 6 days ago (1 children)

It's already permanent and nonstop. They're known to ignore robots.txt, and remove user agent on detection.

And the goal is not only to prevent resource abuse, but break a predatory model.

But, feel free to continue gracefully doing nothing while other takes action, it's bound to help eventually.

[–] [email protected] 27 points 6 days ago (1 children)

There are different kinds of AI scraper defenses.

This one is an active strategy. No shit people know that this costs them resources. The point is that they want to punish the owners of bad-behaved scrapers.

There is also another kind which just blocks anything that tries to follow an invisible link that goes to a resource forbidden by robots.txt

[+] [email protected] -18 points 6 days ago (1 children)

One or two people using this isn't going to punish anything, or make enough of a difference to poison the AI. That's the same phrase all these anti-AI projects for sites and images use, and they forget that, like a vaccine. you have to have the majority of sites using your method in order for it to be effective. And the majority of sysadmins are not going to install what's basically ICE from Cyberpunk on a production server.

Once again, it's lofty claims from the anti-AI crowd, and once again it's much ado about nothing. But I'm sure that won't stop people from believing that they're making a difference by costing themselves money out of spite. 😂

[–] [email protected] 8 points 6 days ago* (last edited 6 days ago) (1 children)

The only AI company that responded to Ars' request to comment was OpenAI, whose spokesperson confirmed that OpenAI is already working on a way to fight tarpitting.

Ah yes. It's extremely common for one of the top companies in an industry to spitefully expend resources fighting the irrelevant efforts of...

One or two people

Please, continue to grace us with you unbiased wisdom. Clearly you've read the article and aren't just trying to simp for AI or start flame wars like a petulant child.

[–] [email protected] -4 points 6 days ago

Well, luckily for them, it's a pretty simple fix. Congrats on being a part of making them jot down a note to prevent tarpitting when they get around to it. You've saved the internet!

And stop pretending like you're unbiased either. We both have our preconceived notions, and you're not more likely to be open to change yours than I am. In fact, given the hysterical hyperventilating anti-AI "activists" get to, we both know you're not ever going to change your mind on AI, and as such you'll glom onto any small action you think is gonna stick it to the man, no matter whether that action is going to have any practical effect on the push for AI or not.

[–] [email protected] 10 points 6 days ago (1 children)

Not like you can load balance requests of the malicious subdirectories to a non-prod hardware. Can be decommissioned hardware.

[–] [email protected] 3 points 6 days ago* (last edited 6 days ago) (1 children)

Serving a pipe from ChatGPT into and AI scraping your site uses little server resources.

[–] [email protected] 369 points 1 week ago (3 children)

They're framing it as "AI haters" instead of what it actually is, which is people who do not like that robots have been programmed to completely ignore the robots.txt files on a website.

No AI system in the world would get stuck in this if it simply obeyed the robots.txt files.

[–] [email protected] 171 points 1 week ago

The disingenuous phrasing is like "pro life" instead of what it is, "anti-choice"

load more comments (2 replies)

[–] [email protected] 84 points 1 week ago (20 children)

I hope it's effective.

load more comments (20 replies)

[–] [email protected] 61 points 1 week ago

It's not that we "hate them" - it's that they can entirely overwhelm a low volume site and cause a DDOS.

I ran a few very low visit websites for local interests on a rural. residential line. It wasn't fast but was cheap and as these sites made no money it was good enough Before AI they'd get the odd badly behaved scraper that ignored robots.txt and specifically the rate limits.

But since? I've had to spend a lot of time trying to filter them out upstream. Like, hours and hours. Claudebot was the first - coming from hundreds of AWS IPs and dozens of countries, thousands of times an hour, repeatedly trying to download the same urls - some that didn't exist. Since then it's happened a lot. Some of these tools are just so ridiculously stupid, far more so than a dumb script that cycles through a list. But because it's AI and they're desperate to satisfy the "need for it", they're quite happy to spend millions on AWS costs for negligable gain and screw up other people.

Eventually I gave up and redesigned the sites to be static and they're now on cloudflare pages. Arguably better, but a chunk of my life I'd rather not have lost.

[–] [email protected] 58 points 1 week ago

Good

[–] [email protected] 53 points 1 week ago* (last edited 1 week ago) (2 children)

I am so gonna deploy this. I want the crawlers to index the entire Mandelbrot set.

I'll train with with lyrics from Beck Hansen and Smash Mouth so that none of it makes sense.

load more comments (2 replies)

load more comments