this post was submitted on 03 Jun 2024
1471 points (97.9% liked)
People Twitter
5383 readers
778 users here now
People tweeting stuff. We allow tweets from anyone.
RULES:
- Mark NSFW content.
- No doxxing people.
- Must be a tweet or similar
- No bullying or international politcs
- Be excellent to each other.
- Provide an archived link to the tweet (or similar) being shown if it's a major figure or a politician.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Do you understand how hard that is? Most websites do not just let you scrape them entirely because they think you're a competitor. Web scraping involves staying one step ahead of each site. You might as well just pick a site and buy most things there.
Plus you can just have everything delivered. It's easier and you can get access to sales.
Yes, I do indeed, for example the AutoTL;DR bot you might have seen around Lemmy (that does web scraping) is mine.
Sure, you can do an entirely different thing than I was talking about, but then you're replying to something entirely else, aren't you? This was a reaction to a post in the OP which doesn't talk about getting everything delivered.
There are 3rd party APIs that handle scraping.... Eg Red circle API. Problem is they don't really have produce and everything up to date.