One of the biggest limitations of BrandQL's early architecture was that all scraping ran from a single server in the US. For American domains, latency was fine. For everything else — European companies, Asian startups, Australian businesses — first-fetch times were painfully slow, sometimes north of 2.5 seconds.
With this release, the entire scraping pipeline now runs on distributed edge workers.
The Problem with Centralized Scraping
When BrandQL scrapes a domain for the first time, it needs to fetch the HTML, parse it, discover image sources, download candidates, and score them. Doing all of that from one location creates two problems:
- High latency for international domains: A request originating in Frankfurt to scrape a Japanese website has to bounce across continents twice
- Single point of failure: If our US server goes down, scraping stops globally
The Solution: Edge Workers
Scrape requests now route to the nearest edge location (powered by Cloudflare Workers). The worker handles the full pipeline — fetch, parse, discover, score — geographically close to the target domain's origin server. Results get cached globally via CDN as before.
Results
The numbers speak for themselves:
- US domains: Average first-fetch dropped from ~1,200ms to ~800ms
- EU/Asia domains: Average first-fetch dropped from ~2,500ms to ~900ms
- Overall reliability: Up from 96% to 99.2%, because geo-distributed requests are less likely to be blocked or time out
The big win is for international domains. Requests that look "local" to the target server get better treatment — fewer CAPTCHAs, fewer timeouts, faster responses.
What's Next
Edge scraping opens the door for more sophisticated orchestration — like retrying failed fetches from a different region, or running parallel scrapes from multiple locations to compare results. That's further down the roadmap, but the foundation is now in place.
These improvements are live for all users. No action needed on your end.