Annotation errors are not a rounding error in the benchmarks the computer vision industry has trusted for a decade. Bounding boxes and class labels are the ground truth that every artificial intelligence vision system is trained on and then graded against, so a mistake in the labels is a mistake the model inherits twice over. A team at Sejong University in Seoul examined roughly 47,000 images drawn from seven of the most widely used object detection datasets, logged every annotation error they could verify, and published the tally. The datasets are the ones on almost every model card: MS-COCO, Open Images, Pascal VOC, LVIS, DOTA, FSOD and Object365.
Tech Xplore covered the work on 7 September 2026 under the headline “Better AI starts with better data.” The underlying paper appeared in Artificial Intelligence Review in February and is open access, which means the numbers behind the press summary are all readable — including the per-dataset breakdown, the sample sizes and the protocol.
That detail matters more than the headline. The press coverage reports the survey’s raw counts. Divide those counts by the sample sizes the paper also states, and the ranking of datasets changes completely: the benchmark with the most logged annotation errors is not the one with the highest annotation error rate. That single arithmetic step is the most useful thing in the paper, and nobody has published it.
This article does three things. It sets out exactly what the Sejong team did and found, it works through the annotation error rates their own figures imply, and it separates what the survey establishes from what it leaves open. Along the way it covers the four families of correction methods the survey catalogues, and what any team training a detector should actually do on Monday morning.
Every figure below is taken from the paper or computed from figures the paper states. Where the paper does not give a number, this article says so rather than estimating one.
Table of contents
- What the Sejong University Annotation Error Audit Actually Did
- The Six Annotation Error Types That Now Have Names
- Annotation Error Counts Across Six Benchmarks
- The Annotation Error Rate Nobody Has Reported
- What Automated Re-annotation Finds Instead
- How an Annotation Error Damages a Trained Model
- Four Ways Teams Fix Annotation Errors
- What This Means If You Train Your Own Detector
- Annotation Error Checks You Can Run This Week
- What the Annotation Error Survey Does Not Settle
- Annotation Error FAQ
- References
What the Sejong University Annotation Error Audit Actually Did
The study is unusual for a survey: it stopped reviewing other people’s work partway through and went and looked at the data itself.
The paper behind the headline
The paper is “Quality over quantity: a data-centric survey of annotation errors in object detection datasets” by Adnan Hussain, Kaleem Ullah, Muhammad Afaq, Muhammad Munsif, Altaf Hussain and Sung Wook Baik, all of the Department of Software at Sejong University. Sung Wook Baik is the corresponding author. It ran in Artificial Intelligence Review volume 59, article 107, published online on 7 February 2026 with the version of record on 13 February.
A survey that did its own fieldwork
Most surveys stop at the literature. This one reviews work published between 2016 and 2025 across 196 references, then adds a manual inspection of the benchmarks themselves. The authors’ stated reason is that annotation quality “receives little attention compared to the creation of algorithms,” so there was no existing count to cite.
Fifteen people, two teams, one script
The inspection protocol is the part worth copying. Fifteen people were split into two sub-teams. Team 1 — ten people — used a Python script to surface candidate annotation errors and recorded each one. Team 2 — five people — verified those candidates. The process then repeated, with both teams working together until every annotation error in a dataset had been found and confirmed.
How the images were chosen
Selection was not uniform. For small, specialised datasets like DOTA the team checked the entire dataset. For large general-purpose ones like Open Images they sampled randomly, but deliberately covered every class. Around 47,000 images went through this process in total, and each confirmed annotation error was logged with its image ID in a supplementary file.
| Element | Detail |
|---|---|
| Institution | Department of Software, Sejong University |
| Journal | Artificial Intelligence Review, vol. 59, art. 107 |
| Dates | Received 11 Aug 2025, accepted 17 Jan 2026, online 7 Feb 2026 |
| Literature covered | 2016 to 2025, 196 references |
| Images inspected | Around 47,000 |
| Inspection team | 15 people, 10 detecting and 5 verifying, iterated |
| Benchmarks inspected | MS-COCO, DOTA, Open Images, Pascal VOC, FSOD, Object365, LVIS |
| Licence | Open access, CC BY 4.0 |
The Six Annotation Error Types That Now Have Names
The survey’s most portable contribution is a standardised taxonomy. Before it, every paper on this subject used its own vocabulary, which made results impossible to compare.
Incorrect labels
The box is in the right place but carries the wrong class, or marks something that is not an object at all. This is the single largest category in the survey’s own count, and it clusters around visually similar or fine-grained classes.
Localization errors
The class is right but the box is wrong: misaligned, too loose, too tight, or clipped. These are the annotation errors the survey singles out as having the greatest effect on training and evaluation, because they degrade quietly rather than obviously.
Inconsistent annotations
The same kind of object is treated differently in different images — an occluded car boxed in one frame and skipped in the next, or a reflection labelled here and ignored there. Consistency failures teach a model a rule that does not exist.
Duplicate annotations
Two or more boxes on one object. Cheap to detect programmatically and, in the survey’s figures, far more common in some datasets than others.
Group errors
A cluster of objects handled wrongly — labelled as one blob when they should be separate instances, or the reverse. In the survey’s count this appears in Open Images alone.
Debatable examples
Cases where competent annotators would genuinely disagree: an object too small, too occluded or too ambiguous to call. The survey counts these separately rather than pretending they are clear-cut, which is the honest treatment.
Missed annotations, the seventh row
The manual protocol used six categories, but the survey’s dataset-by-dataset presence table adds a seventh: missed annotations, meaning a real object with no box at all. It is present in MS-COCO, DOTA, Pascal VOC, Object365 and FSOD, and absent from LVIS and Open Images.
Annotation Error Counts Across Six Benchmarks
The survey publishes a per-dataset, per-type breakdown. Six benchmarks appear in it with numbers attached.
Open Images
The largest logged total. The survey analysed 10,400 samples and identified 240 carrying an annotation error, of which 87 were incorrect labels or false positives and 33 were group inconsistencies. Its conclusion is blunt: even well-structured, large-scale collection still carries substantial noise.
Object365
140 flagged samples, led by 45 inconsistent annotations, then 41 duplicate boxes and 41 labelling errors. The survey attributes this profile to label density and scene clutter — a dataset with 638,000 images and 10.1 million annotations across 365 categories is hard to keep uniform.
LVIS
Across 25,000 samples the team found 112 images carrying an annotation error. LVIS is a long-tailed benchmark built with a strong labelling pipeline, and its profile reflects that: the biggest single group is 40 debatable cases, followed by 37 incorrect labels on visually similar categories.
FSOD
100 flagged samples, of which 41 were labelling errors and 31 were localization errors. The survey flags this as especially serious because few-shot detection is a low-data regime, where a handful of noisy examples can move generalisation a long way.
DOTA
The aerial imagery benchmark. The team checked all 1,869 samples and flagged 51, most commonly 20 incorrect labels and 17 inconsistent annotations. Occlusion, object density and the scale and orientation variation typical of aerial scenes drive most of it.
Pascal VOC
The lowest raw count: 46 flagged from a review of 8,000 images, split fairly evenly with 12 localization errors and 12 inconsistent annotations at the top. Typical cases were misaligned boxes and inconsistent treatment of reflections and occlusions.
| Error type | FSOD | Object365 | LVIS | DOTA | P-VOC | Open Images |
|---|---|---|---|---|---|---|
| Inconsistent annotations | 21 | 45 | 8 | 17 | 12 | 8 |
| Localization errors | 31 | 13 | 24 | 6 | 12 | 14 |
| Incorrect labels | 41 | 41 | 37 | 20 | 10 | 87 |
| Duplicate annotations | 7 | 41 | 1 | 6 | 5 | 4 |
| Group errors | — | — | — | — | — | 33 |
| Debatable examples | — | — | 40 | 2 | 7 | 3 |
| Row total | 100 | 140 | 110 | 51 | 46 | 149 |
The row totals are this article’s arithmetic on the survey’s table, not figures the paper prints. Two of them are slightly below the totals given in the running text — LVIS is described as 112 including two contradictory-labelling cases, and Open Images as 240 including 91 multiple-annotation cases that the table does not break out separately.
The Annotation Error Rate Nobody Has Reported
Here is the step the coverage skipped. The survey states, for four of the six datasets, how many images were inspected as well as how many were flagged.
Why raw counts mislead
Ranked by raw count, Open Images looks like the worst benchmark and Pascal VOC the best. But Open Images was sampled at 10,400 images and LVIS at 25,000, so the two numbers are answering different questions. An annotation error count without a denominator measures how hard somebody looked.
The four datasets with a stated denominator
DOTA: 51 flagged from 1,869 inspected, a rate of 2.73%. Open Images: 240 from 10,400, or 2.31%. Pascal VOC: 46 from 8,000, or 0.57%. LVIS: 112 from 25,000, or 0.45%. Those four rates are simple division on figures the paper states in its own text.
DOTA is the outlier, not Open Images
That reorders everything. DOTA has the fewest logged annotation errors of any dataset with a stated denominator and the highest error rate of all four — roughly one flagged image in every 37. LVIS, which carries more than twice as many logged errors, is the cleanest at about one in 222. The gap between them is a factor of six.
What the rates still cannot tell you
They are rates of flagged images, not of individual boxes, and a single image can carry several annotation errors. They also reflect how thoroughly each dataset was searched: DOTA was inspected exhaustively while the others were sampled, which plausibly raises DOTA’s rate relative to the rest. The direction of that bias is knowable; its size is not.
What Automated Re-annotation Finds Instead
Set the manual figures beside the automated cleanup projects the survey catalogues, and a much larger number appears.
MJ-COCO and the 52,621 images
MJ-COCO is a re-annotated MS-COCO 2017, built with a four-stage pseudo-labelling refinement pipeline to fix missing labels, inaccurate boxes and duplicates without manual work. It keeps the same 123,287 images and 80 categories as the original. Its authors identified 52,621 anomalous images and re-annotated them — 42.7% of the dataset.
Mini6K and the 12% figure
Mini6K is a 6,000-image subset of MS-COCO’s 2017 training split. The survey reports that it contained missing or incorrect labels in over 12% of its images. The cleaned version, Mini6KClean, corrected 729 images and ended up with 55,111 object instances, 10,894 more than the subset it replaced.
Sama-COCO and the re-annotation trend
Sama-COCO is another full re-annotation of MS-COCO 2017, produced by Sama’s in-house team to raise label quality for both detection and segmentation. Together with COCO-OI and MJ-COCO it marks a shift: the response to a benchmark full of annotation errors is now to rebuild it rather than to caveat it.
Why the two numbers disagree
Manual inspection flagged between 0.45% and 2.73% of sampled images. An automated pipeline flagged 42.7% of MS-COCO. These are not contradictory measurements of the same thing. A human logs what they can confirm as wrong; an automated pipeline flags anything its model finds anomalous, including tight judgement calls and its own false positives. The honest reading is that the manual figures are a floor and the automated ones an upper bound, and the true rate sits somewhere between.
How an Annotation Error Damages a Trained Model
The survey devotes a section to consequences, and they are more specific than “worse accuracy.”
Corrupted training signal
An annotation error produces inconsistent gradient updates and unreliable supervision. A missed object teaches the model to penalise a correct prediction; a false box teaches it to find something that is not there. Both push in the wrong direction on every epoch that sees the image.
Distorted evaluation metrics
This is the part that reaches your benchmark table. Missed ground-truth objects convert true positives into false positives and depress precision. Mislabelled and loosely drawn boxes cut recall, and the survey notes the effect is worst at high IoU thresholds above 0.5 — exactly where modern detectors are compared.
Small, occluded and rare objects suffer most
The damage is not spread evenly. It concentrates on small, occluded and partially visible objects, and it is amplified in long-tailed distributions like LVIS and low-data regimes like FSOD, where a few bad examples make up a large share of what the model ever sees of a class.
Dataset-specific shortcuts
Systematic quirks — heavy use of “crowd” labels, inconsistent occlusion handling, category boundaries that shift between annotators — teach shortcuts that hold only inside that dataset. The survey warns this reduces out-of-distribution robustness, which is precisely the risk in safety-critical deployments.
Four Ways Teams Fix Annotation Errors
The survey’s second contribution is a clean categorisation of correction methods into four families, with the trade-off each one makes.
Manual review
Human annotators inspect and correct boxes and classes through iterative verification and consensus. Highest precision, and the only approach that reliably resolves ambiguous instances. It does not scale, and it is subject to fatigue and inter-annotator variation.
Weakly supervised
Models trained on weak labels — image-level tags — infer approximate object locations using class activation maps or saliency cues. It cuts annotation cost and suits large unlabelled pools, but localisation is imprecise and it often finds only the most salient part of an object.
Semi-supervised
A small labelled set plus a large unlabelled pool, combined through pseudo-labelling, consistency regularisation and teacher–student frameworks. A reasonable balance of cost and performance, provided you tune confidence thresholds carefully — the method is sensitive to errors in its own pseudo-labels.
Fully automatic
Model-driven inference, confidence scoring and self-learning detect and correct problems with no human in the loop. It is the only family that scales to continuously updated datasets, and the one most likely to introduce confident false corrections. MJ-COCO sits in this family, which is part of why its flag rate is so much higher.
| Family | Best suited to | Main advantage | Main limitation |
|---|---|---|---|
| Manual | Small sets, high-stakes work such as medical or automotive | Highest precision, resolves ambiguity | Slow, costly, does not scale |
| Weakly supervised | Large unlabelled or low-resource collections | Cheap expansion from image-level tags | Poor localisation, needs refinement |
| Semi-supervised | Projects with a small labelled seed set | Balances cost against performance | Sensitive to pseudo-label errors |
| Fully automatic | Very large or continuously updated datasets | Scales, minimal human input | Can introduce false corrections |
What This Means If You Train Your Own Detector
The survey is about public benchmarks, but the findings transfer directly to private datasets, which are almost always labelled to a lower standard than MS-COCO.
Your benchmark score has an error bar you did not compute
If you report mAP on Pascal VOC or LVIS, some fraction of your false positives are correct detections of objects nobody boxed. The survey’s rates suggest that fraction is small but not zero, and it is largest exactly where model differences are smallest. Two detectors half a point apart may not be distinguishable at all.
Budget a QA pass, not a cleanup
The pattern across every dataset here is that quality control was retrofitted. Building a verification stage into the annotation budget from the start is cheaper than a re-annotation project later, and it is the difference between MS-COCO and MJ-COCO. Teams planning ML model development work should treat label QA as a line item rather than an afterthought.
Borrow the taxonomy
The six categories are the immediately reusable part. Adopting them means your annotators, your reviewers and your scripts describe the same problem the same way, and your defect counts become comparable between projects rather than between people.
Copy the two-team protocol
Separating detection from verification is the cheapest quality lever in the paper. One group surfaces candidates with tooling, a second confirms them, and the loop repeats. It removes the conflict of interest in asking annotators to grade their own work, and it is why the survey’s counts are defensible.
Annotation Error Checks You Can Run This Week
Four of the six categories are detectable with scripts you can write in an afternoon, before any human looks at an image.
Duplicate and near-duplicate boxes
Flag any pair of boxes of the same class whose IoU exceeds roughly 0.9. In Object365 duplicates accounted for 41 of 140 logged annotation errors, so this single check is worth running first on any large dataset.
Class confusions between similar categories
Incorrect labels were the largest annotation error category overall at 236 of 596, concentrated in fine-grained classes. Train a quick classifier on the crops, then review the cases where its confident prediction disagrees with the stored label.
Box tightness
Compare each box against a segmentation or saliency estimate of its contents and flag the loosest and tightest outliers. This surfaces the localization errors that damage evaluation most at high IoU thresholds.
Crowd and group handling
Audit how your dataset treats clusters. Count instances per image per class and inspect the tail — group errors are rare in absolute terms but they cluster in exactly the dense scenes where detectors are hardest to compare.
A sampled human pass
Scripts cannot find the debatable cases or most inconsistent annotation errors. Sample a few hundred images stratified across classes, have two people review them independently, and record disagreements. Feeding the result back into data management and analytics processes turns a one-off audit into a measurable quality baseline.
What the Annotation Error Survey Does Not Settle
Reading the paper rather than the summary makes the limits clear, and they matter for how far the numbers can be pushed.
Two datasets have no stated denominator
FSOD and Object365 are reported with counts — 100 and 140 — but no sample size. The four stated sample sizes add to 45,269 of the roughly 47,000 images inspected, which leaves only about 1,700 images between the two. Either they were sampled very lightly, or “around 47,000” is looser than it reads. No rate should be computed for those two datasets from published figures.
A flagged image is not a quantified rate
The survey counts images and instances its teams confirmed as carrying an annotation error. It does not estimate what proportion of all boxes are wrong, and it does not claim its sampling supports that inference. Treating 2.73% as “DOTA’s error rate” overstates what the method can support.
The debatable category is a judgement call
Forty of LVIS’s 112 flagged samples are debatable examples — more than a third. Reasonable annotators disagree on those, so a different team would produce a different annotation error count. The survey deserves credit for separating them out instead of folding them into the totals.
MS-COCO has no count of its own
MS-COCO appears in the presence table showing which error types occur where, but not in the counted breakdown. Given that MS-COCO is the most-used detection benchmark in existence and the target of three separate re-annotation projects, its absence from the per-dataset annotation error counts is the paper’s most conspicuous gap.
Annotation Error FAQ
What is an annotation error in object detection?
A defect in the ground-truth labels rather than in the model: a wrong class, a badly drawn or missing box, a duplicate, or inconsistent treatment of similar objects. The survey groups them into six named types.
How many images did the researchers check?
Around 47,000, drawn from MS-COCO, DOTA, Open Images, Pascal VOC, FSOD, Object365 and LVIS. Small datasets were checked exhaustively; large ones were randomly sampled with full class coverage.
Which benchmark had the most annotation errors?
By raw count, Open Images, at 240 flagged samples. By rate against the images actually inspected, DOTA at 2.73%, against LVIS at 0.45% — a six-fold spread. The two rankings disagree, which is why the denominator matters.
Do annotation errors mean published detection results are wrong?
No. It means reported scores carry uncertainty that nobody quantifies, and that small gaps between models are less meaningful than they appear. The effect is largest at high IoU thresholds and on rare or small object classes.
Can an annotation error be fixed automatically?
Partly. Fully automatic methods scale and are the only realistic option for very large datasets, but they can introduce confident false corrections. The survey’s own recommendation is a hybrid: automated detection with human verification, which is what its two-team protocol implements.
Where can I read the original paper?
It is open access under CC BY 4.0 in Artificial Intelligence Review, DOI 10.1007/s10462-026-11502-z, with a supplementary file listing every image carrying an annotation error by ID. That supplementary list is the most useful artefact for anyone auditing these datasets.
References
More AI coverage: explore Progressive Robot's AI Models, Tools & Releases hub — hands-on reviews, setup guides and benchmarks in one place.