DataMasters Episode 7, 6 June 2026
Navigating the Grey: Scaling from Single Worker to Multi-VM Undetectable Scrapers
From single scripts to multi-VM fleets: practical techniques to scale undetectable scrapers.
- Organizer
- Data Engineering Pilipinas
- Venue
- Discord
- Attendees
- 20
- Type
- Talk
- Level
- Intermediate
- Duration
- 120 minutes
- web scraping
- Playwright
- curl_cffi
- CAPTCHA
- proxies
- OCR
- asynchronous
- bot evasion
Abstract
A practical guide to scaling web scrapers from a single worker to multi-VM fleets. Covers execution models, scraping frameworks, headed vs headless trade-offs, bot evasion techniques, and real-world challenges like CAPTCHAs and geo-blocking. Includes code samples and a hands-on challenge with seven progressively difficult targets.
Outline
- Execution Models
- Scraping Frameworks
- Headed vs Headless
- Extension & API Scraping
- Agentic Scraping
- OCR Extraction
- Paid Web Searching
- Bot Evasion Techniques
- Scraping at Scale
- Scraping Legalities
- Q&A Session
Key takeaways
- Compare synchronous vs asynchronous scraping and when to use each
- Choose the right scraping framework for your workload (Requests, Scrapy, Playwright, curl_cffi)
- Apply bot evasion techniques: fingerprints, proxies, cookie prewarming, and human-like behavior
- Scale scrapers across multiple VMs while balancing concurrency, success rate, and cost
- Navigate legal and technical obstacles like CAPTCHAs, geo-blocking, and rate limits