This is a most excellent place for technology news and articles.
Yeah, that has like 0 chances for working. At most it would annoy bots for web search, at least it has a proper robots.txt.
But any agent trying to process data for AI is not going to go to random websites. It's going to use a curated list of sites with valuable content.
At this point text generation datasets can be achieved with open data, and data sold by companies like reddit or Microsoft, they don't need to "pirate" your blog posts.
True to a limited extent. Anyone can post a link to somebody's blog on a site like reddit without the blogger's permission, where a web crawler scanning through posts and comments would find it. But I agree with you that a thing like Nepehthes probably wouldn't work. Infinite loop detection is an important part of many types of software and there are well-known techniques for it, which as a developer I would assume a well written AI web crawler would have (although I've never personally made one).
scrape.maxDepth = 5
LOL wow, this is probably the most elegant way to say what I just said to somebody else. Well written web crawlers aren't like sci-fi robots that rock back and forth smoking when they hear something illogical.
What's stopping the sites with valuable content from using this?
A bot that's ignoring robots.txt is likely going to be pretending to be human. If your site has valuable content that you want to show to humans, how do you distinguish them from the bots?
What is robots.txt?
I think sites that feel they have valuable content can deploy this and hope to trap and perhaps detect those bots based on how they interact with the tarpit
This is really interesting.
What a great name!
This showed up on HN recently. Several people who wrote web crawlers pointed out that this won’t even come close to working except on terribly written crawlers. Most just limit the number of pages crawled per domain based on popularity of the domain. So they’ll index all of Wikipedia but they definitely won’t crawl all 1 million pages of your unranked website expecting to find quality content.
Did you read the article? (There is a link to a non walled version.)
Since they made and deployed a proof-of-concept, Aaron B said their pages have been hit millions of times by internet-scraping bots. On a Hacker News thread, someone claiming to be an AI company CEO said a tarpit like this is easy to avoid; Aaron B told 404 Media “If that’s, true, I’ve several million lines of access log that says even Google Almighty didn’t graduate” to avoiding the trap.
Millions of hits may sound like a lot, but you need to view that in context.
What's the context?
The modern internet. Millions of hits is very normal - one of my domains is just 30 year old ASCII art of a penguin, and it gets 2-3 million a month from bots/crawlers (nearly all of them trying common exploits). The idea that the google spider would be notably negatively impacted by this is kinda naive. It could fall fully into the tarpit and it probably wouldn't even get flagged as an abnormal resource allocation. The difference in power between desktop and enterprise equipment is at this point almost inexpressible.
People think of hacking like a thief with a lockpick. It's oftentimes more like someone methodically checking every door in the neighborhood for any that are unlocked.
If it is linked to the Internet then it'll be hit by crawlers. Their "trap" isn't any how many show up but how long each bot stays on their individual site.
Can confirm, I have a website (https://2009scape.org/) with tonnes of legacy forum posts (100k+). No crawlers ever go there.
It's a shame that 404media didn't do any due diligence when writing this
No crawlers ever go there.
if it makes you feel any better, i would go there if i was a web crawler.
Sorry to tell you, but you are indexed at least by duckduckgo, bing, ecosia, startpage, google, and even one of searx' crawlers has payed you a visit.
Then that's a where we hide the good stuff
An even easier way to hide stuff is to not put it online in the first place.
Reminds me of burying folders in folders in folders to hide naughty content as a youth.
I think this rate limiting mechanism is mostly a niceness rule : you should try to not put too much pressure on any website and obey the rules defined in its robots.txt.
So I guess this idea is not bad as it would mostly penalize bad players.
My new favorite is asking if it's cheating to look at your opponent's pieces in chess.
For anybody who ever had this happen, ChatGPT has some solutions to remedy the situation:
I tried the same input and got a more expected answer.
More accurately, it traps any web crawler, including regular search engines and benign projects like the Internet Archive. This should not be used without an allowlist for known trusted crawlers at least.
Just put the trap in a space roped off by robots.txt - any crawler that ventures there deserves being roasted.
More accurately, it traps any web crawler
More accurately, it does not trap any competent crawlers, which have per domain limits on how many pages they crawl.
You would still want to tell the crawlers that obey robots.txt do not pay attention to that part of the website. Otherwise it's just going to break your SEO
This reminds me of that one time a guy figured out how to make "gzip bombs" that bricked automated vuln scanners.
I had a scanner that was relentless smashing a server at work and configured one of those.
evidently it was one of our customers. it filled their storage up and increased their storage costs by like 500%.
they complained that we purposefully sabotaged their scans. when I told them I spent two weeks tracking down and confirmed their scan were causing performance issues on our infrastructure I had every right to protect the experience of all our users.
I also reminded them they were effectively DDOSing our services which I could file a request to investigate with cyber crimes division of the FBI.
they shut up, paid their bill, and didn't renew their measly $2k mrr account with us when their contract ended.
bitch ass small companies are always the biggest pita.
