Back to Changelog

Distributed Scraping on the Edge

First-fetch times are now 30-60% faster, especially for non-US domains. Here's how we did it.

BrandQL's scraping pipeline has moved to the edge. Logo fetching now runs on distributed workers across multiple regions worldwide.

Why edge scraping

Running scrapers from a single server meant high latency for international domains and a single point of failure. Edge workers solve both problems.

What changed

MetricBeforeAfter
Avg first-fetch (US domains)~1,200ms~800ms
Avg first-fetch (EU/Asia)~2,500ms~900ms
Reliability96%99.2%

How it works

Scrape requests are routed to the nearest edge worker. The worker fetches the target domain's HTML, discovers logo sources, and scores them — all close to the origin server. Results are then cached globally via CDN.

Benefits

  • Faster first fetches: Especially for domains hosted outside the US
  • Better reliability: Geo-distributed requests look more "local" to target servers, reducing blocks and timeouts
  • Redundancy: If one region goes down, requests route to the next closest worker

These improvements are live for all users. No changes required on your end.