A practical book on building config-driven web scraping infrastructure. Instead of writing bespoke Python for each scraping task, every job is expressed as a JSON object that a single engine interprets. The same engine handles static pages, JavaScript-rendered content, pagination, scheduling, and change detection.
The final chapters cover integrating AI agents into the loop: generating configs autonomously, running scraping tasks as tool calls via MCP, and closing the feedback cycle so the agent can adapt to layout changes.
Most scraping code is one-off. A site changes its layout and the script breaks. You have fifty scripts across a team and no consistent model for how they work, how they fail, or how to schedule them. Scaling means copying and modifying files.
A config-driven approach treats scraping as a data problem: describe what you want, not how to get it. The engine handles the how.
The config contract is designed from the start to be machine-writable. An LLM can inspect a page and emit a valid JSON spec. That spec feeds into the same engine a human would use. The book walks through building exactly that loop - including the MCP server that exposes the scraper as a set of tools an agent can call directly.
A complete scraping framework: static fetching with httpx, dynamic rendering with Playwright, an auto-fallback mechanism that tries the cheaper path first, a pagination architecture that handles cursor- and offset-based patterns, and a scheduling layer. All wired to a JSON config schema with a Python engine that interprets it.
The book is structured as a working codebase you build chapter by chapter. Every concept ships with runnable code.