verifier.org

Argilla

free open-source annotation tooling with no workforce attached — which is both its strength and the reason it can't finish first.

free#teams-with-their-own-ann#or-open-communities-labe

the only entry with no labour record to examine, because it employs nobody — you supply the people, and their working conditions become your problem rather than a vendor's.

argilla is annotation software rather than an annotation service. hugging face acquired it in june 2024 for a reported $10m, and the core remains free and open source, deployable on your own hugging face space or infrastructure, tightly integrated with the hub.

that architecture answers the data question cleanly — datasets stay under your control, and there is no vendor in the path reusing them — and it removes the pay-transparency problem by removing the workforce. no contractors, no wage floor to publish, no disputes to report.

the description above is ours, condensed from the ranking. pricing moves — check it on the vendor's own page before you rely on it.

pricing
free
our verdict

the only entry with no labour record to examine, because it employs nobody — you supply the people, and their working conditions become your problem rather than a vendor's.

we researched this category against vendors' own pricing pages and licence files. that is where this line comes from — not from the vendor, and not from anything they paid for.

more data labeling vendors

Snorkel AI

we checked thisnot published

reduces how much human labelling you need in the first place, which is the only structural answer to this category's problems.

#teams-with-domain-heavy-#documents-who-would-rath

Labelbox

we checked thisnot published

annotation platform plus an expert marketplace, with the highest reported pay band of the crowd-model vendors.

#teams-wanting-tooling#on-demand-domain-experts#including-medical

Toloka

we checked thisnot published

the largest open crowd here at the lowest reported pay, and a corporate lineage worth tracing before you sign.

#very-high-volume-microta

Surge AI

we checked thisnot published

bootstrapped, profitable and serving the frontier labs — facing a class action over how it classifies the people doing the work.

#frontier-lab-grade-rlhf#reasoning-data-where-qua