Large-Scale Image QC: 500K Assets, 1.5M Multilingual Images

Key Facts

• Scale: 500,000 assets / 1,500,000 images (3 images per asset)
• Dataset: Multilingual, globally sourced, with multiple categories and subcategories
• Delivery: Ad-hoc batches, arriving weekly and bi-weekly
• Platform: Customized, role-based QC platform forked from HaiCrowd
• Approach: Automated QC plus a humans-in-the-loop verification team
• Scalability: Handled all 1,500,000 records seamlessly, with versioned batches and periodic database backups

Client Overview

• Profile: A global organization assembling a large-scale, multilingual image dataset for AI training
• Sourcing: Data collected by many vendors across different countries
• Objective: Turn noisy, vendor-supplied images into a clean, consistent, well-labeled dataset ready for model training

Project Overview

The Challenge

Images were supplied by many different vendors, and the raw data carried a wide range of quality problems that had to be caught before it could be used for training:

1. Duplicate files (MD5 duplicates)

The same image often arrived more than once across different vendor batches. Identical files needed to be detected and removed so the dataset was not skewed by repeats.

2. Similar images and similar backgrounds

Beyond exact duplicates, many images were near-duplicates or shared the same background, which reduces dataset diversity. These needed to be surfaced for a human to judge.

3. Corrupted files - whole-asset rejection

Because each asset is a set of three images that belong together, a single corrupted image makes the whole asset unusable. The pipeline had to reject the entire asset rather than pass a partial set downstream.

4. Ad-hoc, irregular delivery

Data did not arrive on a fixed schedule. Batches came in weekly, bi-weekly and ad-hoc, so QC had to be traceable batch by batch and easy to resume at any time.

5. Typo-filled vendor metadata

The metadata accompanying the images was authored manually by vendors and contained frequent typos, which would have corrupted category and subcategory records if loaded as-is.

6. Multilingual on-image text

The dataset was multilingual and globally sourced, so reviewers frequently faced on-image text in languages they did not read, making verification slow and error-prone with a traditional workflow.

The Solution: a customized QC platform forked from HaiCrowd

HaiData forked its HaiCrowd platform into a QC platform customized for this project, with role-based login so reviewers, supervisors and admins each see the tools and permissions they need. The platform automates the mechanical, repeatable checks and reserves human attention for the judgment calls.

1. Versioned QC

Every batch that arrives is versioned (v1.0, v1.1, v1.2, and so on). This makes the whole QC history traceable, keeps ad-hoc deliveries organized, and lets the team resume, compare or export any specific version at any time.

2. Automated QC on ingestion

As each dataset arrives, the platform automatically:

• Rejects duplicate files using MD5 hashing to catch exact duplicates.
• Rejects corrupted files and, because one asset is three images, rejects the whole asset if any image in it is corrupt.
• Groups similar images into clusters so near-duplicates and shared backgrounds are surfaced together for manual review.
• Corrects metadata typos using fuzzy-matching to normalize vendor-supplied fields against known categories and subcategories before the record enters the database.

3. Manual QC with in-place translation

During manual QC the platform can translate and overlay the on-image text in place, similar to how in-browser translation works, so a reviewer can understand and verify multilingual content regardless of the original language.

4. Cluster-based similarity review

A reviewer can pull up any similarity cluster and visually compare the grouped images side by side, then take the right action on the group. This is far faster and more reliable than trying to spot similar images one at a time.

5. Supervisor QC

After reviewer QC, a supervisor performs a second level of QC on the dataset, providing an independent check before results are considered final.

6. Broad format support

The platform handles all the image formats present in the dataset, including JPG, PNG and HEIC, so no data was left unprocessed because of its file type.

7. Flexible admin export

Admins can export QC results with a range of filters: a specific version, multiple versions combined, or a category-wise slice, so downstream teams get exactly the cut of data they need.

Database backups

Because this is a large, continuously growing dataset, we took periodic full backups of the entire database throughout the project to keep the data safe and recoverable.

Built to scale

The platform was engineered to scale, and handled the full 1.5 million records seamlessly. Automated pre-filtering removed duplicate and corrupted assets on ingestion so only clean data grew the database; every batch was versioned; and QC state for all 500,000 assets stayed fast to query and export as the dataset kept growing week after week. New ad-hoc batches were simply ingested as the next version, with no slowdown and no re-processing of earlier data.

The QC Workflow

The diagram below shows the end-to-end pipeline, from vendor delivery through automated QC, humans-in-the-loop review and supervisor sign-off, to filtered export. Role-based access governs every stage, and the central database is backed up periodically.

Figure: End-to-end QC workflow. Role-based access governs every stage.

Humans-in-the-Loop

Our humans-in-the-loop team quickly understood the client's QC requirements and worked through the dataset on the customized platform, manually verifying the 500,000 assets that passed automated QC. Because automation had already removed duplicates and corrupted assets and grouped the look-alikes into clusters, reviewers could focus their attention where human judgment actually mattered.

Results and Impact

• Scaled seamlessly: The platform handled all 500,000 assets / 1,500,000 records end to end, with no slowdown as ad-hoc batches accumulated week after week.
• Cleaner data, earlier: Duplicates and corrupted assets were removed automatically on ingestion, so bad data never reached manual review.
• Consistent metadata: Fuzzy-matching corrected vendor typos before they entered the database.
• Faster manual QC: Similarity clustering and in-place translation let reviewers work far more efficiently on a global, multilingual dataset.
• Traceable and safe: Versioned batches, filtered exports and periodic database backups kept the growing dataset organized and protected.

By combining automation with a skilled humans-in-the-loop team on a purpose-built platform, HaiData solved a major data-quality problem for the client and saved a substantial amount of manual effort and time.

For more information on how HaiData can help with large-scale data QC and annotation, please write to info@haidata.ai