MS-COCO: Proven Label Error Risk in a Top AI Benchmark

MS-COCO is the benchmark almost every object detector reports against, and a new Sejong University survey has just published a table that flags it for all seven annotation error types it defines. The same survey publishes a second table that counts errors in six benchmarks by hand. MS-COCO is not in it. That combination — every error type ticked, no error counted — is the most interesting thing in a 196-reference paper about data quality in computer vision, and nobody has pointed at it.

This is not a criticism of the authors, and the omission is not hidden. The survey names MS-COCO in its inspection list, gives it the first row of its dataset table, and gives it the first block of its performance table. It simply never publishes a per-category error count for it, the way it does for DOTA, LVIS, Pascal VOC, Open Images, FSOD and Object365. We covered those six counted benchmarks and the error rates they imply in our companion piece on the hidden annotation errors in object detection datasets. This article is about the seventh.

The numbers for MS-COCO exist, but they live somewhere else: in three separate re-annotation projects that each rebuilt the same 123,287 images. Divide their published figures against each other — arithmetic the survey never performs — and two of them converge on the same answer to within three per cent. That is the strongest evidence yet for how much of this benchmark is wrong, and it is assembled from numbers that were already on the page. Anyone whose artificial intelligence roadmap rests on detector benchmarks should know what the arithmetic says.

Table of contents

MS-COCO Is Named in the Inspection and Missing From the Count
What MS-COCO Is, and Why It Sets the Score Everyone Quotes
MS-COCO Scores Seven Out of Seven on the Error Taxonomy
Where the Survey’s Own Counts Do Not Add Up
Three Projects Have Re-annotated the Same 123,287 MS-COCO Images
The MS-COCO Arithmetic Nobody Ran: Boxes Added Per Image
Why Two Independent MS-COCO Estimates Agree to Within 3%
The MS-COCO Instance Totals Disagree With Themselves
Why MS-COCO Was Never Hand-Counted
What an Uncounted MS-COCO Does to Your Benchmark Numbers
How to Work With MS-COCO Knowing All This
What the MS-COCO Question Leaves Open
MS-COCO FAQ
References and Further Reading

MS-COCO Is Named in the Inspection and Missing From the Count

The survey’s Section 3.7 is unambiguous about scope. It says the team examined “DOTA, open images, Pascal VOC, FSOD, Object365, LVIS, and MS COCO” — seven datasets, with MS-COCO listed last but listed. Around 47,000 images were checked in total, by a team of fifteen split into two sub-teams: ten people ran a Python script to surface candidate errors, five verified them, and the loop repeated.

Six subsections for seven datasets

What follows that scope statement is Section 3.7.1 through 3.7.6. FSOD, Object365, LVIS, DOTA, Pascal VOC, OpenImage. There is no Section 3.7.7. The seventh dataset named in the scope sentence never gets its own results paragraph, and the MS-COCO error count is never reported anywhere else in the paper.

Table 4 has six columns, not seven

Table 4 is titled “Manually identified annotation errors across various benchmark datasets” and it is where every counted figure in the survey lives. Its columns are FSOD, Object365, LVIS, DOTA, P-VOC and Open Images. MS-COCO has no column. For the most-cited benchmark in the field, the survey’s only quantitative statement about label quality is an absence.

The dataset is present everywhere else

This is what makes the gap worth writing about rather than shrugging at. MS-COCO is row 1 of Table 6, the dataset description table. It is the first dataset block of Table 5, the performance comparison, with five methods listed against it. It is the first column of Table 7, the error-type comparison. The survey clearly regards MS-COCO as the primary benchmark. It is the only one of the seven it does not count.

What MS-COCO Is, and Why It Sets the Score Everyone Quotes

MS-COCO began as the Microsoft COCO dataset in 2014, with the 2017 revision now in general use. It holds roughly 330,000 images overall; the detection train and validation split is about 123,000 images, with a further ~41,000 test images whose labels are not public. Every image carries bounding boxes and labels across 80 object categories.

The properties that made it the standard

The survey is direct about why MS-COCO displaced Pascal VOC as the default: it contains small objects, dense groups and heavy occlusion, which makes it a harder and more realistic test. Those same three properties — small, crowded, occluded — are precisely the conditions under which human annotators miss objects and misplace boxes.

Why an uncounted benchmark matters more than a counted one

A 2.73% error rate in DOTA affects the aerial-imagery subfield. An unmeasured error rate in MS-COCO affects the headline average precision number in most detection papers published in the last decade. The dataset with the widest blast radius is the one with no published per-category count, and that asymmetry is what this article is about.

Dataset	Venue	Method	Images	Instances	Headline figure
MS-COCO	ECCV 2014	Original crowd annotation	~123K	~850k	123,287 images, 80 categories
Mini6KClean	JVCIR 2023	Manual correction of a subset	6K	~55K	729 images fixed, +10,894 instances
Sama-COCO	arXiv 2023	Professional in-house re-annotation	~123K	~1.1M	+218,000 instances over the original
COCO-OI	ICLR 2023	Merge with Open Images	~398K	~1.5M	80 shared categories, boxes only
MJ-COCO	arXiv 2025	Four-stage automatic pseudo-labelling	~123K	~1.2M	52,621 anomalous images identified

MS-COCO Scores Seven Out of Seven on the Error Taxonomy

Table 7 is the survey’s qualitative companion to Table 4. Instead of counts it carries ticks and crosses: for each of seven error types, does this dataset exhibit it? Seven datasets are compared, MS-COCO first.

The tally nobody added up

Counting the ticks is a thirty-second exercise the paper does not perform. MS-COCO is ticked for all seven — missed annotations, incorrect labels, localization errors, duplicate annotations, inconsistent annotations, debatable examples and group errors. No other dataset scores seven. DOTA, Open Images and Pascal VOC score six each; Object365 and FSOD score five; LVIS scores four.

The dataset with the most error types has the fewest error numbers

Put the two tables side by side and the shape of the gap is clear. Of the 49 cells in Table 7, 39 are ticks. MS-COCO supplies seven of them, the maximum available, and contributes nothing at all to the 596 counted errors in Table 4. It is the only benchmark in the survey flagged for every category of failure and quantified for none of them.

Error type	MS-COCO	DOTA	LVIS	Open Images	P-VOC	Object365	FSOD
Missed annotations	✓	✓	✗	✗	✓	✓	✓
Incorrect labels	✓	✓	✓	✓	✓	✓	✓
Localization errors	✓	✓	✓	✓	✓	✓	✓
Duplicate annotations	✓	✓	✗	✓	✓	✓	✓
Inconsistent annotations	✓	✓	✓	✓	✓	✓	✓
Debatable examples	✓	✓	✓	✓	✓	✗	✗
Group errors	✓	✗	✗	✓	✗	✗	✗
Types flagged	7	6	4	6	6	5	5

Error types flagged in Table 7, out of seven

MS-COCO 7 of 7

DOTA 6 of 7

Open Images 6 of 7

Pascal VOC 6 of 7

Object365 5 of 7

FSOD 5 of 7

LVIS 4 of 7

Counted from Table 7 of the survey. Cells are ticks and crosses; the totals are ours.

Where the Survey's Own Counts Do Not Add Up

Before trusting Table 4 as the complete quantitative record, it is worth checking it against the prose that describes it. Two of the six columns do not reconcile.

LVIS is short by two

Section 3.7.3 reports 112 flagged samples in LVIS. Table 4’s LVIS column holds 8 + 24 + 37 + 1 + 40 = 110. The prose accounts for the difference itself: it mentions “rare cases of contradictory labeling (2)”, a category that has no row in Table 4. The two figures are consistent once you know that, but only the prose tells you.

Open Images is short by ninety-one

The larger gap is Open Images. Section 3.7.6 reports 240 flagged samples. Table 4’s column sums to 8 + 14 + 87 + 4 + 33 + 3 = 149. The missing 91 are described in the prose as “multiple annotation errors”, again a category with no row in the table. That is 38% of the dataset’s flagged samples sitting outside the itemised count.

The totals that follow from this

Add the six prose totals and you get 689 flagged samples. Add the six Table 4 columns and you get 596. The 93-sample difference is exactly the 91 Open Images cases plus the 2 LVIS cases. Neither number is wrong; the taxonomy simply has more categories in the text than it has rows in the table.

One naming inconsistency worth noting

FSOD reconciles perfectly at 100, but its prose uses different labels from the table. Section 3.7.1 describes “bounding box errors (21)” and “annotation errors (7)”, while Table 4’s FSOD column records 21 inconsistent annotations and 7 duplicate annotations. The counts align; the names do not. For a survey whose stated contribution is a standardised error taxonomy, that is a small but real friction.

Dataset	Table 4 column sum	Prose total	Difference	Category with no table row
FSOD	100	100	0	None
Object365	140	140	0	None
LVIS	110	112	2	Contradictory labeling
DOTA	51	51	0	None
Pascal VOC	46	46	0	None
Open Images	149	240	91	Multiple annotation errors
Total	596	689	93
MS-COCO	No column	No subsection	—	All seven types ticked in Table 7

Share of each dataset’s flagged samples that Table 4 itemises

FSOD, Object365, DOTA, Pascal VOC 100%

LVIS 98.2% (110 of 112)

Open Images 62.1% (149 of 240)

MS-COCO no count published

Table 4 column sums divided by the sample totals stated in Sections 3.7.1 to 3.7.6.

Three Projects Have Re-annotated the Same 123,287 MS-COCO Images

The survey does describe MS-COCO’s quality problem — just not with its own numbers. It describes it through other people’s datasets, four of which are derivatives of MS-COCO 2017. Three of those hold the figures this article turns into rates.

Sama-COCO: professional human re-annotation

Sama-COCO was built by Sama’s in-house annotation team, working over the same 123,287 images and the same 80 categories. It decomposes crowd annotations into individual instances and raises the instance count to roughly 1.12 million — which the survey states is 218,000 more than the original MS-COCO 2017. The underlying paper is titled, pointedly, “Benchmarking a benchmark: How reliable is MS-COCO?”

Mini6KClean: manual correction of a 6,000-image subset

Mini6K is a 6,000-image sample drawn from MS-COCO’s train2017 split. Its authors found annotation errors in over 12% of those images, hand-corrected 729 of them, and folded the result back in as Mini6KClean — 6,000 images and 55,111 instances, which is 10,894 more than Mini6K held.

MJ-COCO: automatic four-stage pseudo-labelling

MJ-COCO is the newest of the three and the only fully automated one. A four-stage pseudo-labelling refinement pipeline identified 52,621 anomalous images in MS-COCO and re-annotated them without manual work. Its instance total in Table 6 is roughly 1.2 million.

One dataset, three verdicts, no shared unit

The three projects report in three different currencies: added instances, corrected images, and anomalous images. None of them states an error rate, and the survey never converts between them. That conversion is where the interesting result lives.

The MS-COCO Arithmetic Nobody Ran: Boxes Added Per Image

Two of the three re-annotations state both a count of added boxes and the number of images those boxes were spread across. Dividing one by the other gives a per-image rate that is directly comparable — and neither paper, nor the survey, prints it.

Sama-COCO: 1.77 boxes added per image

218,000 added instances across 123,287 images works out at 1.768 instances per image. That is the average number of objects a professional annotation team found in an MS-COCO image that the original crowd pipeline had not labelled.

Mini6KClean: 1.82 boxes added per image

10,894 added instances across 6,000 images works out at 1.816 instances per image. Different team, different year, different method, a subset one-twentieth the size — and a figure within a rounding error of Sama’s.

The projection, and how close it lands

Scale the Mini6KClean rate up to the full benchmark: 1.816 × 123,287 = 223,847 instances. Sama-COCO actually found 218,000. The projection overshoots by 5,847 boxes, or 2.7%. Two independent MS-COCO re-annotations, using incompatible methods, agree on the size of the missing-label problem to within three per cent.

Why this is a stronger result than either paper claims

Neither project set out to estimate a benchmark-wide error rate; each simply published what it had fixed. The convergence is an artefact of the arithmetic, not a designed replication, which is exactly what makes it persuasive. A shared methodological bias could produce one number twice. Two unrelated pipelines landing 2.7% apart is harder to explain away.

Why Two Independent MS-COCO Estimates Agree to Within 3%

The agreement holds at the per-image level, which is the level that generalises. It does not hold at the image level, and the reason is instructive.

The image-level rates differ by three and a half times

MJ-COCO flagged 52,621 of 123,287 images, which is 42.68%. Mini6K’s team corrected 729 of 6,000, which is 12.15% and matches the paper’s “over 12%” description. Those two figures are 3.51 times apart, on the same dataset, measuring what sounds like the same thing.

Boxes per flagged image explains the gap

Mini6K added 10,894 boxes across 729 corrected images — 14.9 boxes per corrected image. MJ-COCO’s Table 6 instance total of roughly 1.2 million against MS-COCO’s roughly 850,000 implies about 350,000 added boxes across 52,621 flagged images, or 6.7 boxes per flagged image. MJ-COCO flags 3.5 times more images and adds 2.2 times fewer boxes to each.

The two projects are not counting the same thing

That is the resolution. Mini6K’s manual pass targeted the images that were badly wrong and fixed them thoroughly. MJ-COCO’s automatic pass casts a much wider net and catches images with one or two missing objects. Neither is incorrect; they use different thresholds for the word “anomalous”, and no shared definition exists to reconcile them.

Which is why the per-image rate is the number to trust

The instances-per-image figure survives the definitional disagreement because it does not depend on where you draw the anomaly threshold. Both projects added roughly 1.8 boxes for every image in the set, whether they described 12% or 43% of images as faulty. That is the closest thing to a measured MS-COCO error rate currently in the literature.

Share of MS-COCO images each project reports as faulty

MJ-COCO, automatic pipeline 42.7% (52,621 of 123,287)

Mini6K, manual pass 12.2% (729 of 6,000)

This survey’s manual inspection not published

Bars scaled to the largest value. Both percentages are stated counts divided by stated image totals.

The MS-COCO Instance Totals Disagree With Themselves

One caution before anyone builds on these figures: the survey’s own instance counts for the COCO family are not internally consistent, and the inconsistencies are large enough to matter.

A 52,000-box gap on the original

Section 4.10 says Sama-COCO holds roughly 1.12 million instances and that this is 218,000 more than MS-COCO 2017 — which implies an original count near 902,000. Table 6 lists MS-COCO at roughly 850,000 and Sama-COCO at roughly 1.1 million, implying a delta closer to 250,000. The two statements differ by around 52,000 boxes on the same dataset.

A 300,000-box gap on COCO-OI

The same pattern appears one row down. Section 4.11 gives COCO-OI approximately 1.2 million instances; Table 6 gives it approximately 1.5 million. Both figures carry the tilde that marks them as approximate, but a 300,000-instance spread is not a rounding artefact.

What this does and does not undermine

It does not touch the two headline deltas, because 218,000 and 10,894 are stated exactly and each is paired with an exact image count. It does undermine any attempt to express the result as a percentage of MS-COCO’s total boxes, since no reliable total exists in the paper. The per-image rate sidesteps the problem entirely, which is another reason to prefer it.

Why MS-COCO Was Never Hand-Counted

The gap has a plausible and fairly mundane explanation, and it is visible in the author list.

The scale argument

The survey’s method was manual: fifteen people, a Python script to surface candidates, human verification, iterate. Against that protocol, ~47,000 images across six datasets is already a substantial undertaking. MJ-COCO’s pipeline flagged 52,621 anomalous MS-COCO images on its own — more than the survey’s entire manual workload, in one dataset. Hand-counting MS-COCO at the granularity of Table 4 was probably not feasible.

The same lab had already automated it

Three of the survey’s six authors — Muhammad Munsif, Altaf Hussain and Sung Wook Baik — are also authors on the MJ-COCO paper. The MS-COCO numbers were not omitted through oversight; they had already been produced by the same group, using an automated pipeline, and published separately. The survey cites that work in its dataset section rather than repeating it.

Why it still leaves a hole

The explanation is reasonable and the outcome is still awkward. Table 4 invites cross-dataset comparison, and the one benchmark most readers care about cannot be compared, because its numbers were produced by a different method and reported in a different unit. A footnote stating that MS-COCO was excluded from the manual count and why would have closed the gap at no cost.

What an Uncounted MS-COCO Does to Your Benchmark Numbers

The practical consequence is not that detectors are secretly bad. It is that a slice of the reported difference between detectors is noise from the labels rather than signal from the models.

Missing boxes punish the models that find them

The survey’s Section 3.8 is explicit: unlabelled ground-truth objects turn correct predictions into false positives, which depresses precision. A detector that genuinely finds the 1.8 objects per image the original pipeline missed is scored down for doing so. The better the recall, the worse the penalty.

The effect concentrates at high IoU

Localization errors and misplaced boxes hit hardest at the stricter intersection-over-union thresholds. Since MS-COCO’s headline metric averages precision across IoU from 0.5 to 0.95, label noise is weighted into exactly the region where modern detectors are separated by fractions of a point.

Small, crowded and occluded objects carry most of it

The properties that make MS-COCO a good benchmark are the properties that make it hard to annotate. Group annotations, occlusion handling and dense scenes are where the survey’s error taxonomy concentrates, and they are where model rankings are least stable.

Why a 0.3-point improvement should not be trusted alone

None of this invalidates a large, replicated gain. It does mean that a fractional improvement measured only on MS-COCO validation, with no ablation and no second benchmark, sits inside the noise floor that 1.8 missing boxes per image implies. Teams building detection into products should treat MS-COCO leaderboard deltas as directional, not decisive.

How to Work With MS-COCO Knowing All This

None of the above argues for abandoning the benchmark. It argues for a few cheap habits that most teams do not currently have.

Evaluate on a cleaned variant as well as the original

Sama-COCO, Mini6KClean and MJ-COCO all exist and all cover the same images. Running your evaluation twice — once on MS-COCO 2017 and once on a re-annotated variant — costs one extra pass and tells you immediately whether a result depends on the labels. A gain that survives both is worth more than a gain that only appears on one.

Report the annotation version you used

“Evaluated on COCO” is now ambiguous. State the year, the split and whether the labels are the original release or a re-annotation. This is the same discipline any serious approach to data management and analytics already applies to every other dataset in a pipeline.

Audit your own data with the survey’s taxonomy

The seven error types in Table 7 are a usable checklist for in-house datasets, whatever the domain. Missed annotations, incorrect labels, localization errors, duplicates, inconsistency, debatable cases and group errors cover most of what goes wrong. Sampling a few hundred images against those seven headings is a day of work.

Budget for verification, not just annotation

The survey’s own protocol had a five-person verification team checking a ten-person detection team. Most in-house labelling budgets assume one pass and no verification. If a professionally annotated public benchmark still needed 218,000 corrections, a first-pass internal dataset needs a second pair of eyes before it trains anything. Our ML model development work treats that verification pass as part of the build, not an optional extra.

Keep a held-out set you have checked yourself

The cheapest protection against benchmark noise is a small evaluation set your own team has annotated and verified, drawn from the data you actually operate on. A few hundred images, checked twice, will tell you more about production behaviour than a fractional gain on a public leaderboard.

What the MS-COCO Question Leaves Open

The survey advances the field and the gap it leaves is worth stating plainly rather than pretending the arithmetic here closes it.

No per-category breakdown exists

The 1.8-boxes-per-image figure is a missing-annotation rate. It says nothing about how many MS-COCO labels are wrong rather than absent, how many boxes are misaligned, or how many are duplicated. Table 4 answers those questions for six datasets and for MS-COCO nobody has.

The re-annotations have not been cross-checked

Sama-COCO, Mini6KClean and MJ-COCO have never been compared against each other on the same images. Whether they flag the same objects or different ones is unknown, and it determines whether 1.8 per image is a floor or an estimate of the whole.

The test split is untouched

Every figure discussed here comes from the train and validation splits. MS-COCO’s ~41,000 test images have no public labels, so nobody outside the benchmark’s maintainers can audit them at all — and they are what the leaderboard is scored on.

The automated pipelines need their own error rate

MJ-COCO corrected 52,621 images with no human in the loop. A pseudo-labelling pipeline has a false-positive rate of its own, and the survey does not report one. Automated correction is the only method that scales to this benchmark, which makes its reliability the open question that matters most.

MS-COCO FAQ

Does the survey say MS-COCO is the worst benchmark?

No. It ticks MS-COCO for all seven error types in Table 7, which is more types than any other dataset, but “exhibits a type of error” is not the same as “has a high rate of it”. Without counts, no ranking is possible.

Where do the 1.77 and 1.82 figures come from?

They are the survey’s own stated numbers divided by its own stated image counts: 218,000 ÷ 123,287 for Sama-COCO, and 10,894 ÷ 6,000 for Mini6KClean. The division is ours; both inputs are the paper’s.

Is MS-COCO still worth using?

Yes. It remains the most widely supported benchmark in object detection and the only one with a decade of comparable results behind it. The recommendation is to evaluate against a cleaned variant alongside it, not to replace it.

Which re-annotation should a team pick?

Sama-COCO for human-verified precision, MJ-COCO for the broadest automated coverage, Mini6KClean if a 6,000-image subset is enough. Running two of them and comparing is more informative than picking one.

Is 42.7% of MS-COCO really wrong?

That is MJ-COCO’s count of anomalous images, and its threshold is loose — an image with one unlabelled object counts. The manual pass on Mini6K put the figure at 12.2%. The honest answer is that it depends entirely on what you count as an error, which is the underlying problem.