Distributed scraping platform
Distributed scraping across 55+ fashion domains (Zara, ASOS, Net-a-Porter, Farfetch, …) plus 5 social platforms, backed by a 50M-capacity Bloom filter and an adaptive-concurrency Go browser pool: the content firehose the whole product runs on.
Each domain is its own service module (scraper · parser · worker · writer · state · proxy), all extending a shared BaseScraper abstraction, with 10+ microservices running in parallel on Kubernetes. Bloom-filter deduplication sits at 50M capacity and a 0.0001 false-positive rate, hydrated from PostgreSQL in 10K-row batches. Queue backpressure kicks in at 800K depth and resumes at 500K. PostgreSQL COPY-protocol bulk inserts (10–100× faster than executemany) handle the write side, with a 3-tier fallback for anything COPY can't swallow.
The image-embedding pipeline is its own distributed system: a producer scans Postgres and enqueues via Redis Lua atomic scripts; workers pull lease-based items with a 300s TTL, process 50 concurrent images per pod, and ship to S3 and Zilliz in parallel (5K-per-batch writes, 10 concurrent). Bidirectional reconciliation jobs detect and fix PG ↔ Zilliz drift on a schedule so the indices never diverge.
The browser side runs on Go: Playwright over CDP with adaptive concurrency that scales sessions on system memory (down at 90%, up at 80%), an 8-instance proxy pool with LRU eviction and sticky reuse, and 10 proxy configs spanning datacenter, residential, and ISP providers. Resource blocking for everything we don't care about keeps each page light.