196

16441 readers

1525 users here now

Be sure to follow the rule before you head out.

Rule: You must post before you leave.

^other^ ^rules^

founded 1 year ago

MODERATORS

[email protected]

321

rulebots.txt (lemmy.world)

submitted 2 months ago by [email protected] to c/[email protected]

34 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 9 points 2 months ago (1 children)

Perhaps, likely depends on the crawler though

[–] [email protected] 12 points 2 months ago

Yeah i dont think ignoring robots.txt is even illegal. They can ofcourse just block your crawlers IP but that would be a cat and mouse game that they would lose in the end.