- Home
- Skills
- Data & Databases
- LLM Dataset Annotator
More screenshots
Works with the AI Agents you already use
LLM Dataset Annotator
Your agent labels every row of a CSV, JSONL or Excel file against a written spec, checks every batch, measures agreement, scores against your own labels and hands back a short human review queue.
$5
LLM Dataset Annotator
Example session with this skill installed
Label every review in reviews.csv with an overall sentiment (Positive, Negative, Mixed, Neutral) and the topics it mentions (Quality, Price, Delivery, Support, Usability). I hand-labeled 20 of them in gold.csv, so check yourself against those first.
- Read your context and instructions
- Compiled the llm dataset annotator
- Generated the data export
Labeled all 40 reviews against the spec we agreed (a definition for every label, six decision rules for the hard cases, three worked examples). Both batches passed validation. Your file is untouched; the labels are in reviews_annotated.csv.
Checked against your 20 gold labels
| Field | Rows | Score |
|---|---|---|
| sentiment | 20 | accuracy 100%, kappa 1.00 |
| topics (multi-label) | 20 | exact-set accuracy 95%, mean Jaccard 0.97 |
The one topics miss is
r28 ("Oh great, another lamp that dies right after the return window closes"): you tagged it Quality + Support, I tagged Quality only. It's a judgment call on whether the return window is a Support complaint; worth a rule either way.
Label distribution
| sentiment | count | topic | count | |
|---|---|---|---|---|
| Negative | 17 (42.5%) | Quality | 19 (47.5%) | |
| Positive | 15 (37.5%) | Usability | 12 (30.0%) | |
| Mixed | 4 (10.0%) | Price | 7 (17.5%) | |
| Neutral | 4 (10.0%) | Support | 6 (15.0%) | |
| Delivery | 5 (12.5%) |
Confidence: 33 high, 7 medium, 0 low.
Review queue: 2 rows (review.csv)
| id | review | my label | why it's flagged |
|---|---|---|---|
| r21 | "Assembly instructions were missing a step, but it went together eventually." | Neutral | minor complaint that resolved itself; Neutral vs Negative is a judgment call |
| r26 | "I ordered white and received black. Exchange was quick though." | Positive | rule 3 (resolved problem praised) says Positive, but the reviewer gives no overall verdict; could be Mixed |
Output columns added
sentiment, topics (joined with ; ), annotation_confidence, annotation_agreement, needs_review, annotation_note.
Next steps
- Settle r21 and r26 in
review.csv(fillfinal_sentiment), then I'll re-export. - Decide the r28 rule (is a return-window complaint also Support?) and I'll add it to the spec.
- For the full dataset, I'll also run the independent second pass on a 300-row sample to report agreement.
reviews_annotated.csv
CSV · data export
Example file from a real run - the skill writes it into your workspace.
Connects securely to your tools. The creator never sees your data.
About this skill
Asking an AI agent to "label this CSV" usually gets you one of two things: a quick keyword script pretending to be judgment, or a pile of labels with no way to tell which ones are shaky. This skill makes your agent work like a careful annotation team instead: it writes the labeling rules down with you, labels every row by actually reading it, checks its own work, and tells you exactly which rows still need a human.
No API key and no extra cost: the agent you already use does the reading. A bundled Python script (standard library only) does the bookkeeping, so no row is skipped, duplicated or given a label that isn't in your spec.
What it does
- Writes the label spec with you: the fields, a definition for every label, decision rules for the hard cases and worked examples. Already have a codebook? It converts it.
- Pilots first on 30-50 rows, or on the rows you labeled yourself, and turns its mistakes into sharper rules before the full run.
- Labels every row in validated batches. Invented labels, broken rules, missing or duplicated rows and bad output are rejected with the exact problem named, and the agent fixes them before anything counts.
- Measures consistency by re-labeling a sample in an independent second pass (a fresh subagent where your agent supports one): percent agreement, Cohen's kappa and the label pairs it confuses most.
- Scores against your gold labels: accuracy, kappa, and precision, recall and F1 per label, with the exact rows to look at.
- Builds a review queue of disagreements, low-confidence rows and the agent's notes. Labels you correct there flow into the export.
- Gives your file back intact: label columns plus confidence, agreement and review status appended; every original column, value and row unchanged. It never overwrites the input.
How it works
- Inspect: columns, row count, empty cells, text lengths, duplicates.
- Spec: agree the labels with you; the script validates the spec against the data.
- Pilot: label a sample or your gold rows, score, refine the rules.
- Pass 1: label every row in shuffled batches; each batch is checked on submit. Runs can be paused and resumed.
- Pass 2: independent re-label of a sample for agreement.
- Report: label distribution, agreement, confusions, review queue.
- Review and export: you settle the flagged rows; the annotated file is written.
Use cases
- Sentiment, emotion, stance, intent or topic labels for NLP training and evaluation sets.
- Qualitative coding of open-ended survey answers or interview excerpts against a codebook.
- Triage of support tickets, feedback, app reviews or moderation queues.
- Extending a small human-labeled set to the rest of your data, with accuracy measured on the part you labeled.
Single-label and multi-label fields. CSV (comma, semicolon, tab or pipe, BOM-safe), TSV, JSONL and Excel.
Known limitations
- Agreement between two passes of the same model measures consistency, not correctness. For accuracy, label 30-100 rows yourself and use the gold comparison; the report says this too.
- It is as fast as your agent reads: hundreds of rows are comfortable; for many thousands, expect a long session or label a sample.
- Labels come from fixed label sets. Span highlighting and free-text answers are out of scope.
Agreement between two passes of the same model measures consistency, not correctness; use your own gold labels for accuracy. Speed depends on your agent: hundreds of rows per session are comfortable, many thousands take a long time. Labels come from fixed label sets; span highlighting and free-text answers are not supported. Excel files need openpyxl.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 5 days ago
Requires
Permissions
Runs one bundled Python script locally (standard library only) to read your data file and write the labeled copy next to it. It makes no network calls, reads no environment variables and never modifies the input file.