All workCatalog collector

Site down in the middle of a run.

A robot goes through a whole online catalog and delivers a spreadsheet, a CSV and a PDF report with charts. It runs on a public site made for scraping practice, so no data belongs to a client.

Status: demo
Period: Sep 2026
Stack: Python 3.12, async Playwright, Pydantic v2, openpyxl, reportlab, matplotlib, pytest

Done by me

Collection, validation and exports in Python.

Context

Public training site, no client data.

Inside

Temporary or permanent

429, 5xx and timeout are temporary: a growing wait with a little randomness, so several robots do not come back together. 4xx is permanent and exits right away. Trying a 404 again only wastes someone else's site.

Python

for attempt in range(1, self.s.attempts + 1):
    try:
        if self._gate is not None:
            await self._gate.wait()
        resp = await page.goto(url, wait_until=self.s.wait_until,
                               timeout=self.s.nav_timeout_ms)
        status = resp.status if resp else 0
        if status == 429 or status >= 500:
            raise TransientHTTPError(f"HTTP {status}")
        if status >= 400:
            raise PermanentHTTPError(f"HTTP {status}")
        return status, attempt
    except PermanentHTTPError:
        raise
    except (TransientHTTPError, PlaywrightTimeoutError, PlaywrightError) as exc:
        last_exc = exc
        if attempt == self.s.attempts:
            break
        wait = self.s.backoff_base_s * (2 ** (attempt - 1)) + random.uniform(0, 0.3)
        await asyncio.sleep(wait)
raise last_exc

Four robots, one pace

When robots.txt asks for an interval, a global gate with a lock spaces the start of each request. Without it, four robots would hit four times faster than requested. The training site does not ask for an interval, so this gate only ran in tests. One detail: the Windows timer wakes up about 15 ms early, so the wait is a loop, not a single sleep.

Delivery you can schedule

Exit codes for a scheduler, delivered files swapped together so nobody picks up a half-written one, protection against formulas in CSV, and an event log in JSONL. Each record goes through validation: price above zero, rating from 1 to 5.

How I know it works

In the real run on September 25, 2026: 1,000 products, 50 categories, 80 pages, 0 failures, 0 invalid records, in 23.6 s of collection. That run had 0 retries, so the growing wait was proven in an offline test, with simulated responses in a real Chromium.

What this does not do

It is a demo, on a training site. The prices are not from a real store.
In that run the robots.txt of the training site answered 404: following its rules was exercised in the tests.