Navigating the Grey: Scaling from Single Worker to Multi-VM Undetectable Scrapers

Abstract

Modern web scraping requires navigating strict bot defense systems like Akamai and reCAPTCHA. Extracting data reliably at scale demands an architecture that accurately mimics human behavior for automated problem resolution. This session outlines the technical requirements for building resilient, high throughput scraping pipelines capable of operating undetected over time.

Outline

1. Building: where the data lives, from static HTML and Playwright to the page's own API
2. Renting or running search and scraping, and levels of OCR
3. Performance: sync and async, concurrency and parallelism, threads, processes, and the GIL
4. Scaling out: one job queue, containers, and a Terraform-managed fleet of VMs
5. Surviving: robots.txt, rate limits, reading failures, and a canary that fails loudly
6. Undetected: what a site checks, stealth browsers, proxies and IP reputation, extension scraping, burner emails, and CAPTCHA solvers
7. The greynet challenge: a practice target website

Key takeaways

• Building: look before you scrape, trying the page's own API, then the HTML, then a browser
• Performance: overlap the waiting, use processes for parsing, then scale out with one image, one queue, and many VMs
• Surviving: start low, back off on 429, and check the content, because a 200 can mean no
• Undetected: every layer must agree, with a clean IP, a real browser, and a human pace