Data Annotation Services: Is 99% Accuracy Enough?

A dataset can pass an impressive quality score and still contain problems that matter to your AI project.

Perhaps the common examples are labelled correctly while rare cases are repeatedly missed. Perhaps reviewers agree with one another because they are following the same unclear instruction. Or perhaps the labels look correct in the annotation tool but arrive in a format your team cannot use.

This is why choosing data annotation services requires more than comparing an accuracy percentage, delivery schedule and price.

The important question is:

“What was measured, against which standard, and what did the review miss?”

Before outsourcing your next dataset, use this guide to evaluate whether the proposed quality process matches the work your model needs to learn.

1. Ask What “99% Accuracy” Actually Measures

An accuracy claim is incomplete without a definition.

Does it describe correct image categories? Accepted bounding boxes? Individual attributes? Reviewed records? Agreement between annotators?

Those are different measurements.

Consider a hypothetical dataset containing 1,000 items:

• 990 items belong to a common category.
• 10 items belong to an important rare category.
• Every common item is labelled correctly.
• Every rare item is labelled incorrectly.

The overall item-level accuracy is 99%. Accuracy on the rare category is 0%.

This arithmetic example is not a provider’s performance result. It demonstrates why an overall score can conceal a concentrated problem.

Before accepting a quality target, ask the provider to explain:

• The unit being scored.
• The reference used to judge correctness.
• The sample size and selection method.
• Results by category or relevant subgroup.
• How missing annotations are counted.
• How ambiguous cases are handled.

A percentage becomes useful when you understand what sits behind it.

2. Define the Task Before Negotiating the Volume

“Label 50,000 images” describes a quantity. It does not define the job.

The same images might require classification, object detection, segmentation or several attributes per object. Each requires different instructions and review criteria.

Start with the intended use.

What should the model distinguish? Which mistakes matter most? What information must the annotator record?

For example, a product-image project may require category labels, visible packaging attributes or object boundaries. Those tasks should not be treated as interchangeable.

Write down:

Brief component	What to specify
Data type	Images, text, audio, video or another format
Annotation task	What the worker must identify or mark
Label definitions	Meaning of each category and attribute
Inclusion rules	What should receive an annotation
Exclusion rules	What should be ignored
Uncertain cases	When to flag, skip or request a decision
Delivery requirements	File structure, identifiers and required fields

OURS GLOBAL lists image, text, video, audio, 3D/LiDAR and custom annotation services. Match your enquiry to the actual task rather than requesting a generic rate for “AI data.”

3. Treat Annotation Guidelines as Part of the Deliverable

Instructions should do more than list category names.

They should explain how to make decisions when examples are incomplete, overlapping or ambiguous.

Suppose an object is partly hidden. Should the annotator mark only the visible area or estimate the full object boundary?

Either approach may be appropriate for a particular task. Mixing them without a rule creates inconsistency.

For text, define how to handle abbreviations, overlapping entities and uncertain sentiment. For audio, clarify how to represent unintelligible speech or overlapping speakers. For video, explain when an object identity should continue across frames.

Include examples showing:

• A clear positive case.
• A clear negative case.
• A difficult boundary case.
• A common mistake.
• A situation requiring escalation.

Then test whether different annotators interpret the instructions consistently.

If repeated questions appear, improve the guideline before increasing production volume.

4. Agreement Between Annotators Is Useful—but It Is Not Proof of Truth

Having several people review the same item can reveal disagreement and support quality control.

AWS documentation describes annotation consolidation as a way to combine multiple worker responses, with methods varying by task. This is a concrete example of using multiple annotations rather than relying solely on one worker’s answer.

However, agreement and correctness are different questions.

If everyone follows an incorrect definition, they may agree on the wrong label. If a task is subjective, disagreement may reflect genuine ambiguity rather than careless work.

Ask how the provider separates:

• Annotator consistency.
• Agreement with approved reference labels.
• Unclear instructions.
• Legitimately ambiguous examples.

Define who makes the final decision when reviewers disagree. For specialised work, that decision may require a qualified subject-matter expert from your organisation or an agreed review team.

Do not let unresolved decisions disappear into a majority vote without considering the task.

5. Make the Pilot Representative of the Difficult Work

A pilot containing only straightforward examples gives limited evidence about production quality.

Include the conditions your team expects to encounter:

• Rare categories.
• Low-quality source material.
• Partially visible objects.
• Similar-looking classes.
• Unusual phrasing.
• Background noise.
• Missing context.
• Examples that challenge the current instructions.

Keep the pilot manageable enough to inspect carefully.

Review both the labels and the questions raised during annotation. A thoughtful clarification request can reveal an issue in the brief before it spreads across thousands of records.

Agree on what must be resolved before scaling:

1. Label definitions are understood.
2. Difficult cases have a decision process.
3. Quality checks are measurable.
4. Output files pass your import checks.
5. Rework responsibilities are clear.

A pilot should test the workflow, not simply produce a polished demonstration.

6. Review Rare Cases Separately From Overall Quality

Random sampling can provide useful information about a batch, but important low-frequency cases may receive little attention in a small sample.

Ask for a review plan that reflects the dataset’s risks and composition.

For example, consider separately reviewing:

• Classes with few examples.
• Categories frequently confused with one another.
• Items flagged as uncertain.
• Work completed under a new guideline version.
• Batches where error rates changed.
• Data from a new source or capture environment.

Keep targeted review results distinct from an overall estimate. A deliberately difficult review sample should not be presented as though it represents the whole dataset without explanation.

The goal is to understand where quality is strong and where further work is needed.

7. Define How Incorrect Labels Will Be Corrected

Quality review is incomplete if it identifies problems without specifying what happens next.

Ask the provider how corrections move through the workflow.

Will rejected items return to the original annotator? Will another reviewer adjust them? Who checks the corrected version?

AWS documentation distinguishes label verification from label adjustment: one evaluates existing annotations, while the other supports correcting them. That distinction is useful when specifying an outsourced review process.

Your agreement should cover:

• What qualifies as a rejected item.
• How feedback is recorded.
• Which corrections are included.
• Expected rework timing.
• How recurring errors are investigated.
• Whether a guideline change requires relabelling earlier batches.

Also distinguish a provider error from a change in your requirements. Both may require work, but they should not be confused.

8. AI-Assisted Labelling Still Needs a Review Plan

If a provider uses model-generated labels as a starting point, ask how people review those suggestions.

Clarify which items receive human attention and how accepted labels are checked.

Useful questions include:

• Are all suggested labels reviewed or only a selection?
• Can reviewers reject or replace a suggestion?
• Are difficult examples routed differently?
• How are repeated model errors detected?
• Are manually created and automatically suggested labels distinguishable where needed?

Do not assume a human-in-the-loop label means every item received the same depth of review.

Equally, do not assume manual work is automatically error-free. Evaluate the process and its evidence rather than choosing by label alone.

9. Validate the Export Before Approving the Dataset

A correctly annotated example is not useful if its identifier points to the wrong source file.

Test an early export using the workflow that will consume the production data.

Check:

• File and record identifiers.
• Category names and IDs.
• Coordinate conventions where applicable.
• Required attributes.
• Missing values.
• Duplicate records.
• Text encoding.
• Version information.
• Compatibility with your expected schema.

For geometry-based tasks, inspect imported examples visually as well as checking file structure.

Do this during the pilot. Discovering an incompatible convention after final delivery can create avoidable rework.

Ask the provider to preserve traceability between the source item, annotation and review status.

10. Compare Cost Per Accepted Deliverable

A low rate per item may exclude review, adjudication, corrections or export preparation.

Compare providers using the same task definition and acceptance standard.

A useful cost discussion should address:

Cost component	Question to ask
Initial labelling	What counts as one billable unit?
Complexity	Do objects, attributes or duration affect the rate?
Quality review	What review is included?
Disagreement resolution	Who handles difficult decisions?
Corrections	Which rework is included?
Guideline updates	How are scope changes priced?
Export	Are validation and custom formatting included?

Also consider your internal review time. A cheaper batch that requires extensive correction may not be the most useful purchase.

Avoid assuming a higher price guarantees better quality. Use the pilot and documented acceptance criteria to compare the actual work.

What to Send a Data Annotation Provider

Prepare an initial project brief containing:

1. The model’s intended task.
2. Data types and estimated volume.
3. A representative, appropriately shareable sample.
4. Draft label definitions.
5. Known difficult cases.
6. Required output format.
7. Quality and acceptance criteria.
8. Review responsibilities.
9. Delivery milestones.
10. Access and data-handling requirements.

Before transferring sensitive or confidential data, agree on authorised access, approved tools, storage locations, retention and deletion arrangements. Confirm whether subcontractors or external AI tools will be involved.

These decisions belong in the project setup, not as assumptions discovered during production.

Frequently Asked Questions

Is 99% annotation accuracy always good?

It depends on how the score is calculated and where the remaining errors occur. Review the metric definition, reference standard, sample design and category-level results.

What is the difference between annotation quality and model accuracy?

Annotation quality concerns how well labels meet the agreed standard. Model performance depends on additional factors, including data coverage, training choices and evaluation design. A high annotation score does not guarantee a particular model result.

Should every item be reviewed by multiple people?

Not necessarily. The appropriate process depends on ambiguity, task complexity, risk and budget. Agree on where multiple reviews or expert decisions add value.

Can we change label definitions after the project begins?

Yes, but changes should be documented and versioned. Determine which earlier annotations are affected and whether they need correction before combining batches.

What is the best way to begin outsourcing data annotation?

Start with a representative pilot, clear guidelines and an export test. Resolve recurring questions before expanding the workload.

Discuss Your Data Annotation Project With OURS GLOBAL

A strong annotation project starts with a definition of quality that both teams can apply.

You need clear labels, representative examples, a process for disagreement and a delivery format your team can use.

Explore OURS GLOBAL’s data annotation and labelling services and share your task, data type, expected volume and quality requirements.

Ask the team to propose a pilot covering annotation, review, corrections and export validation before agreeing on larger production volumes.

Request a project quotation through the service page or email info@oursglobal.com.

Before accepting “99% accurate,” ask what the remaining 1% contains.