Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ

54772 readers

411 users here now

1. Posts must be related to the discussion of digital piracy

2. Don't request invites, trade, sell, or self-promote

3. Don't request or link to specific pirated titles, including DMs

4. Don't submit low-quality posts, be entitled, or harass others

💰 Please help cover server costs.


Ko-fi	Liberapay

founded 1 year ago

MODERATORS

1168

submitted 1 year ago by psychothumbs@lemmy.world to c/piracy@lemmy.dbzer0.com

68 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] pootriarch@poptalk.scrubbles.tech 4 points 1 year ago

It exists, it's called a robots.txt file that the developers can put into place, and then bots like the webarchive crawler will ignore the content.

the internet archive doesn't respect robots.txt:

Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes.

the only way to stay out of the internet archive is to follow the process they created and hope they agree to remove you. or firewall them.