Navigating the Grey: Building Resilient Python Scrapers That Survive at Scale

Abstract

A talk built on scraping the store directories of Philippine malls. It shows the three places data can live (static HTML, a JavaScript page, the page's own API), takes a 53-mall run from 32.5 s to 2.0 s with asyncio and one process per core, then covers rate limits, proxies, monitoring that fails loudly, and how to decide which barriers to stop at.

Outline

1. What a web scraper is and where it sits in the data value chain
2. Three doors to the data: static HTML, a JavaScript page, the page's own API
3. Reading robots.txt before the page
4. Where the time goes: sync, async, threads, and processes across 53 malls
5. Rate limits and getting more per request
6. What a site can see: IP, JavaScript, and proxies
7. Containerizing the browser stack
8. PengePassportPH: built-in limits and a canary that fails loudly
9. Navigating the grey: what a barrier protects and who pays

Key takeaways

• Look before you scrape: the page's own API, then the HTML, then a browser
• Overlap the waiting, then use every core: asyncio for the waiting, one process per core for the parsing
• They set the bar: start low, back off on errors and Retry-After, and get more per request
• Fail loudly: a 200 can mean no, so check every answer and run a check on a schedule
• In the grey, ask who pays: what does the barrier protect, and would you sign your name to it?