Chasing Ghosts: How fighting bots turned one facet hack into a threat-scoring engine
Video Description
speaker: bburg
AI crawlers and aggressive scrapers now generate the majority of traffic to many public-sector Drupal sites — and the old defenses don't hold. User-Agent blocklists are trivially spoofed, and per-IP rate limiting is useless against distributed operations that rotate real-looking browser identities across hundreds of thousands of residential IPs. On one high-traffic federal `.gov` site, automated traffic accounted for nearly 59% of scored requests, over 12 million blocked requests in a matter of months.
Bot Blocker is a lightweight, dependency-free Drupal module that takes a different approach: instead of trying to enumerate every bad bot by name, it scores each anonymous request on how faithfully it reproduces a real browser's request fingerprint. A composite threat score is assembled from many independent header and behavioral signals, and a three-tier User-Agent "scope gate" automatically exempts honest, self-identifying crawlers (Googlebot, Bingbot, search.gov, and friends) while reserving full scrutiny for clients impersonating a browser.
This session is the story of how the module evolved into 2.0. The headline change is architectural: detection was rebuilt as a ThreatDetector plugin system, where every signal is a small, weighted, independently testable plugin. Detection logic became composable and extensible without ever touching the core scorer. We'll trace how analyzing a single real 24-hour scraping event — millions of requests from ~878,000 unique IPs, a suspiciously flat pool of rotated Chrome/Safari User-Agents exhaustively paging through listing views, directly produced new behavioral detectors that catch what header inspection alone misses. We'll also cover making the block response pluggable (a configurable 403/410 page or a fully themeable Twig template) and adding an operations dashboard that surfaces live block metrics and the signals driving them.
Just as valuable are the hard-won deployment lessons: how Bot Blocker behaves behind a CDN and Varnish, how it interacts with Drupal's internal page cache, why a stateless, per-request design is the right layer for this problem (and why per-IP rate limiting is futile against fan-out botnets), how to keep the whole thing dependency-free for easy adoption, and how we used kernel tests and config-schema validation to ship changes with confidence.
If you run a Drupal site that's drowning in bot traffic, or you just want a practical case study in designing an extensible, plugin-based subsystem in Drupal, this talk is for you.
## What attendees will learn
- Why browser-fingerprint scoring beats name-based blocklists and IP rate limits against modern AI scrapers
- How to design a weighted, plugin-based detection system in Drupal that's composable and testable
- How to turn real access-log analysis into targeted detection signals
- Practical patterns for pluggable responses, operational dashboards, and config validation
- Real numbers and deployment gotchas from running this on a federal `.gov` behind Acquia/Varnish
AI crawlers and aggressive scrapers now generate the majority of traffic to many public-sector Drupal sites — and the old defenses don't hold. User-Agent blocklists are trivially spoofed, and per-IP rate limiting is useless against distributed operations that rotate real-looking browser identities across hundreds of thousands of residential IPs. On one high-traffic federal `.gov` site, automated traffic accounted for nearly 59% of scored requests, over 12 million blocked requests in a matter of months.
Bot Blocker is a lightweight, dependency-free Drupal module that takes a different approach: instead of trying to enumerate every bad bot by name, it scores each anonymous request on how faithfully it reproduces a real browser's request fingerprint. A composite threat score is assembled from many independent header and behavioral signals, and a three-tier User-Agent "scope gate" automatically exempts honest, self-identifying crawlers (Googlebot, Bingbot, search.gov, and friends) while reserving full scrutiny for clients impersonating a browser.
This session is the story of how the module evolved into 2.0. The headline change is architectural: detection was rebuilt as a ThreatDetector plugin system, where every signal is a small, weighted, independently testable plugin. Detection logic became composable and extensible without ever touching the core scorer. We'll trace how analyzing a single real 24-hour scraping event — millions of requests from ~878,000 unique IPs, a suspiciously flat pool of rotated Chrome/Safari User-Agents exhaustively paging through listing views, directly produced new behavioral detectors that catch what header inspection alone misses. We'll also cover making the block response pluggable (a configurable 403/410 page or a fully themeable Twig template) and adding an operations dashboard that surfaces live block metrics and the signals driving them.
Just as valuable are the hard-won deployment lessons: how Bot Blocker behaves behind a CDN and Varnish, how it interacts with Drupal's internal page cache, why a stateless, per-request design is the right layer for this problem (and why per-IP rate limiting is futile against fan-out botnets), how to keep the whole thing dependency-free for easy adoption, and how we used kernel tests and config-schema validation to ship changes with confidence.
If you run a Drupal site that's drowning in bot traffic, or you just want a practical case study in designing an extensible, plugin-based subsystem in Drupal, this talk is for you.
## What attendees will learn
- Why browser-fingerprint scoring beats name-based blocklists and IP rate limits against modern AI scrapers
- How to design a weighted, plugin-based detection system in Drupal that's composable and testable
- How to turn real access-log analysis into targeted detection signals
- Practical patterns for pluggable responses, operational dashboards, and config validation
- Real numbers and deployment gotchas from running this on a federal `.gov` behind Acquia/Varnish