Running a Large-Scale Web-Scraping Aggregator in Production

Ever considered scraping data from various top-tier sources to power your own product? It looks straightforward — a great business idea to dive into.

Think again. Here we share the real challenges and the sophisticated solutions involved in making it work at scale, in production — based on a system our team designed, built, and operated for a client.

Context and Motivation

Over the years we've taken on many projects — from small to large-scale — that involve scraping data from various sources to power chatbots, websites, and platforms across automotive, real estate, marketing, and e-commerce. Most technical blogs give general recommendations across sources of varying complexity, but they rarely show concrete, long-term techniques for dealing with these challenges day to day in production. This write-up aims to fill that gap with a real example and concrete practices.

Drawing on a project we delivered for a client working with well-known titans in the automotive industry, we'll walk through the large-scale production challenges of a system built on these sources. This includes:

• Handling page structure changes
• Avoiding IP bans
• Overcoming anti-spam measures
• Addressing fingerprinting
• Staying undetected / hiding scraping behavior
• Maximizing data coverage
• Mapping reference data across sources
• Implementing monitoring and alerting systems

We'll also touch on the legal challenges and considerations related to data scraping.

About the Project

The project — which we designed, built, and ran for our client — is a web-based, distributed microservice aggregator that gathers car offers from the most popular sources across CIS and European countries. It powers advanced analytics that answer critical questions in the automotive market, including:

• The most profitable way and path to buy a car at any given moment, factoring in currency exchange rates, global market conditions, and other variables.
• Whether it's more advantageous to buy a car from another country or on the internal market.
• The average time it takes to sell a specific car model in a particular country.
• Price trends across different regions.
• How economic and political changes impact car sales and prices.

The system maintains and updates a database of around 1 million active car listings and stores historical data since 2022 — over 10 million listings in total, enabling comprehensive data collection and detailed analysis. This extensive dataset helps the client's users make informed decisions with valuable market insights and trends.

High-Level Architecture Overview

Microservices. The system is composed of multiple microservices, each responsible for specific tasks such as data listing, storage, and analytics. This modular approach lets each service be developed, deployed, and scaled independently. The key microservices are:

• Cars Microservice: handles the collection, storage, and updating of car listings from various sources.
• Subscribers Microservice: manages user subscriptions and notifications, keeping users informed of updates and relevant analytics.
• Analytics Microservice: processes the collected data to generate insights and answer key questions about the market.
• Gateway Microservice: the entry point for all incoming requests, routing them to the appropriate microservices while managing authentication, authorization, and rate limiting.

Data Scrapers. Distributed scrapers gather listings from various sources, designed to handle page structure changes, avoid IP bans, and overcome anti-spam measures like fingerprinting.

Data Processing Pipeline. Collected data flows through a pipeline of cleaning, normalization, and enrichment, ensuring it's consistent and ready for analysis.

Storage. A combination of relational and non-relational databases stores current and historical data, allowing efficient querying and retrieval of large datasets.

Analytics Engine. An advanced engine processes the data using machine-learning algorithms and statistical models to generate insights.

API Gateway. Handles all incoming requests and routes them to the appropriate microservices, while managing authentication, authorization, and rate limiting.

Monitoring and Alerting. A comprehensive system tracks the performance of each microservice and overall system health. It's configured with numerous notifications to monitor scraping behavior, so any issue or anomaly — including changes in page structure and potential anti-scraping measures — is detected and addressed promptly.

Challenges and Practical Recommendations

Below are the challenges we faced building this platform and the practical recommendations we implemented to overcome them. These insights come from real production experience and are meant to give you actionable strategies for similar problems.

Challenge: Handling page structure changes

Overview. One of the most significant challenges in web scraping is handling changes in the structure of web pages. Websites often update their layouts — for aesthetics or UX — and these changes can break scrapers that rely on specific HTML structures.

Impact. When a site changes its structure, scrapers can fail to find the data they need, leading to incomplete or incorrect collection. That severely impacts data quality and any insights derived from it, rendering the analysis ineffective.

Recommendation 1 — Leverage API endpoints. To handle frequent structure changes, we shifted from scraping HTML to leveraging the underlying API endpoints used by the web applications (which, admittedly, isn't always possible). By inspecting network traffic, then identifying and testing API endpoints, we achieved more stable and consistent extraction. Finding the right endpoint and parameters can take anywhere from an hour to a week. In some cases we logically deduced endpoint paths; in the best cases we discovered GraphQL documentation by appending /docs to the base URL.

Recommendation 2 — Utilize embedded data structures. Some modern web apps embed structured data within their HTML using structures like __NEXT_DATA__. This can also be leveraged to handle page structure changes effectively.

Recommendation 3 — Define required properties. To control data quality, define the required properties that must be fetched to save and use the data downstream. Attributes vary from source to source, so it's critical to define what's required based on your domain model and future usage. We used the Template Method pattern to dictate how and what attributes are collected during parsing, ensuring consistency across all sources and all parser types (HTML, JSON):

namespace Example
{
    public abstract class CarParserBase<TElement, TSource>
    {
        protected ParseContext ParseContext;

        protected virtual int PinnedAdsCount => 0;
        protected abstract string GetDescription(TElement element);
        protected abstract IEnumerable<TElement> GetCarsAds(TSource document);
        protected abstract string GetFullName(TElement element);
        protected abstract string GetAdId(TElement element);
        protected abstract string GetMakeName(TElement element);
        protected abstract string GetModelName(TElement element);
        protected abstract decimal GetPrice(TElement element);
        protected abstract string GetRegion(TElement element);
        protected abstract string GetCity(TElement element);
        protected abstract string GetSourceUrl(TElement element);

        // more attributes here

        private protected List<ParsedCar> ParseInternal(TSource document, ExecutionContext executionContext)
        {
            try
            {
                var cars = GetCarsAds(document)
                .Skip(PinnedAdsCount)
                .Select(element =>
                {
                    ParseContext = new ParseContext();
                    ParseContext.City = GetCity(element);
                    ParseContext.Description = GetDescription(element);
                    ParseContext.FullName = GetFullName(element);
                    ParseContext.Make = GetMakeName(element);
                    ParseContext.Model = GetModelName(element);
                    ParseContext.YearOfIssue = GetYearOfIssue(element);
                    ParseContext.FirstRegistration = GetFirstRegistration(element);
                    ParseContext.Price = GetPrice(element);
                    ParseContext.Region = GetRegion(element);
                    ParseContext.SourceUrl = GetSourceUrl(element);

                    return new ParsedCar(
                        fullName: ParseContext.FullName,
                        makeName: ParseContext.Make,
                        modelName: ParseContext.Model,
                        yearOfIssue: ParseContext.YearOfIssue,
                        firstRegistration: ParseContext.FirstRegistration,
                        price: ParseContext.Price,
                        region: ParseContext.Region,
                        city: ParseContext.City,
                        sourceUrl: ParseContext.SourceUrl
                    );
                })
                .ToList();

                return cars;
            }
            catch (Exception e)
            {
                Log.Error(e, "Unexpected parsing error...");
                throw;
            }
        }
    }
}

Recommendation 4 — Dual-parser approach. Where possible, cover each source with two parser types — HTML and JSON (via direct API access). Place them in priority order and implement a chain-of-responsibility so one falls back to the other if the HTML or JSON structure changes. This buys a window to update the parsers, at the cost of double the maintenance. We also implemented rotating priority and the ability to dynamically change or remove a parser's priority in the chain via metadata in storage — adjustments without redeploying the whole system.

Recommendation 5 — Integration tests. Integration tests are crucial, even just for local debugging and quick issue identification. Especially when something breaks in the live environment and logs aren't enough, these tests are invaluable. Ideally they live in the CI/CD pipeline — but if a source requires a proxy or advanced techniques to fetch data, maintaining them inside CI/CD can get overly complicated.

Challenge: Avoiding IP bans

Overview. Avoiding IP bans is critical when scraping large volumes from multiple sources. Websites implement anti-scraping measures to detect and block IPs that behave suspiciously — such as making too many requests in a short period.

Impact. When an IP is banned, the scraper can't access the target, resulting in incomplete collection. Frequent bans disrupt the process, cause data gaps, and can halt the whole operation — affecting the quality and reliability of the data that analysis and decisions depend on.

Common causes of IP bans:

1. High request frequency — sending too many requests in a short period.
2. Identical request patterns — repetitive requests that deviate from normal user behavior.
3. Suspicious user-agent strings — outdated or uncommon user agents that raise suspicion.
4. Lack of session management — failing to manage cookies and sessions appropriately.
5. Geographic restrictions — accessing the site from regions the target flags or restricts.

Recommendation 1 — Distribute via cloud services. Cloud functions like AWS Lambda, Azure Functions, or Google Cloud Functions help avoid bans: they have native time triggers, scale out well, run across a range of IPs, and can be located in regions close to the source's real users. This distributes the load and mimics genuine user behavior.

Recommendation 2 — Leverage different proxy types. A variety of proxies helps distribute requests and reduce ban risk:

• Datacenter proxies — Pros: fast, affordable, widely available. Cons: easily detected and blocked due to their non-residential nature.
• Residential proxies — Pros: real residential IPs, much harder to detect and block. Cons: more expensive and slower.
• Mobile proxies — Pros: mobile-carrier IPs, high anonymity, low detection. Cons: the most expensive, and potentially slower due to mobile network speeds.

Mixing these types lets you better distribute requests and reduce the likelihood of detection and banning.

Recommendation 3 — Use scraping services. Services like ScraperAPI, ScrapingBee, Bright Data and similar platforms handle much of the heavy lifting around rotating IPs, managing user agents, and avoiding detection. They can be expensive, though — on this project we often exhausted a whole month's plan in a single day due to high data demands. They're best when budget allows and requirements fit within service limits. We also found that the most complex sources, with advanced anti-scraping mechanisms, often didn't work well with such services.

Recommendation 4 — Combine approaches. Use all of the above sequentially, from the lowest to the highest cost, via a chain-of-responsibility (the same pattern we used for parser types). This allows a flexible, dynamic combination of strategies — all stored and updated as metadata in storage, enabling efficient, adaptive scraping.

Recommendation 5 — Mimic user traffic patterns. Scrapers should hide within typical user traffic based on time zones: more requests during the day, near-zero at night. We split the parsing frequency into 4–5 tiers:

• Peak load
• High load
• Medium load
• Low load
• No load

This reduces the chance of detection and banning. Here's an example of parsing frequency across a typical day for two environments:

Challenge: Overcoming anti-spam measures

Overview. Anti-spam measures prevent automated systems from overwhelming servers or collecting data without permission. They can be sophisticated — user-agent analysis, cookie management, and fingerprinting.

Impact. These measures block or slow scraping, resulting in incomplete collection and longer acquisition times — hurting efficiency and effectiveness.

Common anti-spam measures:

• User-agent strings — sites inspect them to tell a legitimate browser from a known scraping tool. Repeated requests with the same user agent get flagged.
• Cookies and session management — sites track sessions and behavior via cookies; an automated-looking session can be terminated or flagged.
• TLS fingerprinting — details from the SSL/TLS handshake form a unique fingerprint; differences can indicate automation.
• TLS version detection — automated tools may use outdated or uncommon TLS versions, which can be used to identify and block them.

Complex real-world reactions we hit:

• Misleading IP-ban messages. We received "too many requests from your IP" messages when the actual issue was missing cookies for fingerprinting. We spent considerable time troubleshooting proxies before realizing the IPs weren't the problem.
• Fake data return. Some sites counter scrapers by returning slightly altered data — e.g. a car's mileage listed as 40,000 km when the real value is 80,000 km — making it hard to tell whether the scraper is even working correctly.
• Incorrect error reasons. Servers sometimes return misleading error messages, sending troubleshooting down the wrong path.

Recommendation 1 — Rotate user-agent strings. Rotate through a variety of legitimate user agents to simulate different browsers and devices, making it harder to detect and block based on user-agent patterns.

Recommendation 2 — Manage cookies and sessions. Handle cookies as a real browser would — storing and reusing them across requests and managing expiration — to maintain continuous, believable sessions.

Real-world solution. On one source, fingerprint information was embedded in the cookies. Without that specific cookie, it was impossible to make more than 5 requests in a short period without a ban. We discovered those cookies could only be generated by visiting the site's main page with a real/headless browser and waiting 8–10 seconds for it to fully load. Running Selenium for every request was impractical for our volume, so instead we ran multiple Docker instances with Selenium that continuously visited the main page, mimicked authentication, and harvested fingerprint cookies. We then used those cookies in high-volume scraping via HTTP API calls — rotating them with other headers and proxies. That let us make up to 500,000 requests per day, bypassing the protection.

Recommendation 3 — Implement TLS-fingerprinting evasion. Mimic the SSL/TLS handshake of a legitimate browser by configuring common cipher suites, TLS extensions, and versions that match real browsers. This article covers the topic well.

Real-world solution. One source started returning fake data because of TLS-fingerprinting issues. To fix it, we built a custom proxy in Go to modify parameters like cipher suites and TLS versions so our scraper appeared as a legitimate browser. It required deep customization of the SSL/TLS handshake to avoid detection. Here's a good Go example.

Recommendation 4 — Rotate TLS versions. Support multiple TLS versions and rotate between them; using the latest versions common in modern browsers helps blend in with legitimate traffic.

Challenge: Maximizing data coverage

Overview. Maximizing coverage ensures the scraped data represents the most current, comprehensive information available. A common approach is to fetch listing pages ordered by creation date — but at peak times, new offers can appear faster than the scraper can parse the listing pages, leaving gaps.

Impact. Missing new offers produces incomplete datasets, hurting the accuracy and reliability of downstream analysis, missing insights, and reducing the effectiveness of the application.

Problem details:

• High volume of new offers — at peak times, new offers can exceed the scraper's real-time capacity.
• Pagination limitations — listing pages often cap pagination, making it hard to retrieve everything when volume is high.
• Time sensitivity — new offers must be captured as soon as they're created to keep data fresh and relevant.

Recommendation — Use additional filters. Split data by categories, locations, or parameters such as engine or transmission type. Segmenting lets you increase parsing frequency per filter category — a targeted approach that scrapes more efficiently and ensures comprehensive coverage.

Challenge: Mapping reference data across sources

Overview. Mapping reference data is crucial for consistency and accuracy when integrating multiple sources — a common problem in domains like automotive and e-commerce, where sources use different nomenclature for the same entities.

Impact. Without proper mapping, data from different sources becomes fragmented and inconsistent, hurting the quality and reliability of analytics and leading to misinterpretations.

Automotive domain. Inconsistent naming: one source calls a model "Mercedes-Benz V-Class," another "Mercedes V classe." Attribute variation: engine types, transmission types, and trim levels also differ across sources.

E-commerce domain. Inconsistent categories: one platform uses "Electronics > Mobile Phones," another "Electronics > Smartphones." Attribute variation: brand names, specs, and tags differ across sources.

Recommendation 1 — Create a reference-data dictionary. Build a comprehensive dictionary of all possible names and variations as the central repository for mapping to a standardized set of terms. Use fuzzy matching during collection to accurately map similar terms to the standard.

Recommendation 2 — Use image detection and classification. When critical attributes (like a car model's generation) aren't always available textually, image detection and classification can identify them. ML models trained to recognize makes, models, and generations from photos help fill the gaps when text is incomplete or inconsistent. This dramatically reduces manual work and constant mapping updates — but it adds architectural complexity, increases infrastructure cost, and can lower throughput, affecting real-time freshness.

Challenge: Implementing monitoring and alerting

Overview. Effective monitoring and alerting is crucial for a scraping system's health and performance — detecting issues early, reducing downtime, and keeping collection running smoothly. Scraping-specific concerns include detecting changes in source sites, handling anti-scraping measures, and maintaining data quality.

Impact. Without proper monitoring, issues go unnoticed — incomplete collection, more downtime, and significant impact on data-dependent applications. Good monitoring means timely detection and resolution, keeping the system reliable.

Recommendation — Real-time monitoring. Track performance and status in real time. Use dashboards to visualize key metrics — successful requests, error rates, data volume, cookie-generation rates, and succeeded/failed HTTP calls per source per instance — so problems surface immediately. We also ran daily notification reports per source so the team could spot anomalies at a glance.

A Funny Story to End On

Our system scraped continuously from many sources, which made it highly sensitive to any downtime or change in a site's accessibility. There were numerous cases where our monitoring detected that a website was down or unreachable from certain regions. Several times, our team contacted the site's support to tell them that "user X from country Y" couldn't access their site.

In one memorable case, our automated alerts flagged an issue at 6 AM: a popular car-listing service was inaccessible from several European countries. We reached out to their support with the details. The next morning they thanked us and said they'd resolved it — it turned out we had notified them before any of their own users did.

Final Thoughts

Building and maintaining a large-scale web-scraping system is not an easy task. It means dealing with dynamic content, overcoming sophisticated anti-scraping measures, and ensuring high data quality. It's naive to think parsing data from various sources is straightforward — the reality is constant vigilance and adaptation, and real cost in both infrastructure and continuous effort. But with the practices above, you can build a robust, efficient system that handles whatever these sources throw at it.

This is exactly the kind of system we build and operate for clients — and the fallback pipeline described here is baked into our own scraping engine.

Want to build something like this? Read the Multi-Channel Auto Aggregator success story, or book a call to talk through your use case.

Uladzislau Kuzmich — Software Architect & Co-Founder at Pinobyte · LinkedIn