Excerpt from a message I just posted in a #diaspora team internal f...
pod.geraspora.de
Excerpt from a message I just posted in a #diaspora team internal forum category. The context here is that I recently get pinged by slowness/load spikes on the diaspora* project web infrastructure (Discourse, Wiki, the project website, ...), and looking at the traffic logs makes me impressively angry. In the last 60 days, the diaspora* web assets received 11.3 million requests. That equals to 2.19 req/s - which honestly isn't that much. I mean, it's more than your average personal blog, but nothing that my infrastructure shouldn't be able to handle. However, here's what's grinding my fucking gears. Looking at the top user agent statistics, there are the leaders: 2.78 million requests - or 24.6% of all traffic - is coming from Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot). 1.69 million reuqests - 14.9% - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonb...

Evidence for the DDoS attack that bigtech LLM scrapers actually are.

  • raoul@lemmy.sdf.orgEnglish
    17·
    2 days ago

    The only simple possibles ways are:

    • robot.txt
    • rate limiting by ip
    • blocking by user agent

    From the article, they try to bypass all of them:

    They also don’t give a single flying fuck about robots.txt …

    If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

    It then become a game of whac a mole with big tech 😓

    The more infuriating for me is that it’s done by the big names, and not some random startup. Edit: Now that I think about it, this doesn’t prove it is done by Google or Amazon: it can be someone using random popular user agents

    • jherazob@fedia.io
      5·
      2 days ago

      I do believe there’s blocklists for their IPs out there, that should mitigate things a little