Reddit third-party client ban closed user messages behind paywall. I think we the Lemmitors should stop AI training on us or at least monetise it (for our instances)

  • redrum@lemmy.ml
    link
    fedilink
    arrow-up
    5
    arrow-down
    2
    ·
    2 months ago

    Instances could add this snippet to theirs robots.txt (source: Eff.org, businessinsider.com and nytimes.com/robots.txt ):

    User-agent: GPTBot
    Disallow: /
    
    User-agent: Google-Extended
    Disallow: /
    
    User-agent: Meta-ExternalAgent
    User-agent: meta-externalagent
    Disallow: /
    

    Note: this only tell to the crawlers of openai, google and meta to not crawl the site to traiN a LLM, the nytimes have a large list of other crawlers.