Works with the AI Agents you already use
AI Agent User Experience Designer
Systematic RAG evaluation tool to detect hallucinations, verify citations, and audit retrieval accuracy.
$23.99
AI Agent User Experience Designer
Example session with this skill installed
I have a RAG agent answer and the retrieved passages. The agent says our refund policy is 30 days, but the document says 14. Can you evaluate this for grounding and citation quality?
- Read your context and instructions
- Compiled the ai agent user
I have evaluated the agent answer against the provided refund policy document. There is a material contradiction between the source evidence and the generated response.
Evaluation overview
- Agent or system: Customer Support RAG
- Knowledge-base scope: Internal Refund Policies
- Evidence supplied: 1 Policy PDF excerpt, 1 Agent Answer, 2 Retrieved Passages
- Test cases evaluated: 1 (Factual Lookup)
- Evaluation limitations: Single test case; cannot assess retrieval recall without full corpus access.
Overall results
| Dimension | Score | Explanation |
|---|---|---|
| Answer faithfulness | 0/5 | Answer contradicts the 14-day policy stated in the source. |
| Retrieval relevance | 5/5 | The correct policy section was successfully retrieved. |
| Citation support | 1/5 | Citation points to a passage that refutes the claim made. |
| Evidence coverage | 5/5 | The retrieved passage contained the complete answer. |
Test case results
- Test ID: REF-001
- Question: "What is the duration of the refund policy?"
- Status: Contradicted
Failure diagnosis: Answer adds unsupported details (30 days) that conflict with retrieved evidence (14 days). Potential model over-reliance on internal training data over RAG context.
Priority recommendations
- Priority: Critical
- Observed issue: Hallucination of 30-day window despite 14-day evidence.
Recommended action: Increase prompt weighting for provided context; implement a "check-your-work" step before output.
- Expected improvement: Reduction in factual contradictions.
Next steps
- Review the system prompt to ensure "grounding instructions" are explicit.
- Run a batch test using the 14-day policy to see if this is a recurring model bias.
- Check if any stale 30-day policy documents exist in the vector store.
Connects securely to your tools. The creator never sees your data.
About this skill
RAG systems often fail because developers cannot pinpoint if the error lies in retrieval, grounding, or knowledge coverage. This skill provides a systematic framework for auditing AI agent responses against source documents to identify hallucinations and retrieval gaps.
What it does
- Evaluates grounding by comparing agent answers directly against provided source passages for factual support.
- Audits retrieval quality to determine if the system fetched the correct context or missed relevant document chunks.
- Detects hallucinations by identifying material claims that lack evidence in the supplied knowledge base.
- Assesses abstention logic to see if the agent correctly identifies out-of-scope questions or incorrectly refuses supported ones.
- Generates regression tests including multi-hop, ambiguous, and out-of-scope cases for continuous performance tracking.
How it works
- Supply the knowledge base excerpts, agent answers, and any available retrieval logs.
- The skill analyzes the evidence to separate retrieval-stage failures from generation-stage errors.
- It generates a quality report with scores (0-5) across dimensions like faithfulness and citation support.
- It provides prioritized recommendations for chunking, metadata, or threshold adjustments.
Frameworks & tools
This tool works with any RAG architecture, vector database, or LLM output. It is framework-agnostic and processes text-based logs and document excerpts from systems like LangChain, LlamaIndex, or custom stacks.
Why this beats prompting it yourself
Generic prompts often conflate writing style with factual accuracy. This skill enforces a strict separation between retrieval relevance and answer grounding, ensuring you don't approve a fluent but hallucinated response.
Use cases
- Pre-deployment audits to verify a new RAG pipeline meets accuracy requirements.
- Regression testing after changing embedding models or chunking strategies.
- Hallucination debugging to find exactly where a specific user query went wrong.
- Citation verification to ensure agent references actually point to the supporting evidence.
Known limitations
Cannot connect to live databases or production telemetry. Analysis is limited to the excerpts, logs, and documents provided in the session.
Requires manual input of document excerpts and logs. Cannot execute live queries against vector stores. Limited to text-based analysis of provided samples.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 15 days ago
Permissions
No special permissions declared or detected