fastcompany.com

Meta and Alibaba AI Crawlers Nearly Take Down LGBT History Project Archive

Meta Platforms (Facebook) logoMeta Platforms (Facebook)
Sig2.85Sen-4.75Env0.00Soc-3.65Gov-4.60
Alibaba logoAlibaba
Sig2.45Sen-2.85Env0.00Soc-2.55Gov-2.95

The LGBT History Project, which reached 50 million views since 2011 and has been archived by the British Library, was nearly taken offline after AI bots began scraping it. The volunteer operator moved hosting to AWS, but crawlers tied to Alibaba and others drove heavy traffic for weeks, forcing Cloudflare and bot-blocking work. He says one set of crawlers pulled about 12GB and that Meta bots ignored robots.txt.

Coverage (1)

ABOUT

What is Fimmer?

Most feeds treat all company headlines the same. An executive interview, an office closure, stock analysis, and a data breach can sit side by side with no sense of scale. Fimmer ranks coverage by what matters, so important events stand out and signals are prioritised over noise. We cluster related company news and score each story using a range of metrics, so you see the most crucial stories first.

SCORING

What do the scores mean?

Significance (0–10) measures the scale of the real-world impact. Sentiment and ESG (−10 to +10) indicate the tone of the coverage and whether the change relates to environmental, social, or governance factors. Together, these scores help distinguish meaningful developments from routine company news. More information about our scoring methodology can be found on our About page.

DISCOVERY

Explore more on Fimmer

Search more than 10,000 public companies, browse the company index, explore their scores, and read the coverage behind them, with our core features available to everyone for free.

For more advanced features, Pro gives you access to dynamic feeds, custom sorting, and more ways to track the companies that matter to you.

More with Pro