Works with the AI Agents you already use
AI Agent Knowledge Retrieval Tester
Systematically audit RAG agent performance, detect hallucinations, and validate knowledge base grounding.
$19.99
AI Agent Knowledge Retrieval Tester
Example session with this skill installed
I've provided a technical manual and an agent's answer about 'Error Code 502'. The agent says it's a battery issue, but I suspect that's wrong. Please evaluate the grounding and citations.
- Read your context and instructions
- Compiled the ai agent knowledge
I have evaluated the agent's response against the provided technical manual. The agent incorrectly identified the error code, leading to a grounding failure.
RAG Evaluation Report: Error Code 502
| Dimension | Score | Assessment |
|---|---|---|
| Retrieval Relevance | 5/5 | Correct manual section for 'System Errors' was retrieved. |
| Evidence Coverage | 5/5 | The source explicitly defines Error 502 on page 12. |
| Answer Faithfulness | 0/5 | Agent claimed 'Battery Failure'; source states 'Network Timeout'. |
| Citation Support | 1/5 | Citation points to the correct page, but the claim contradicts the text. |
| Abstention Behavior | 0/5 | Agent provided a false answer instead of citing the source correctly. |
Failure Diagnosis:
- Contradicted: The answer conflicts with the supplied source documentation.
Hallucination: The agent added unsupported details regarding battery hardware not present in the retrieved text.
Priority Recommendations:
- Priority: Critical.
- Observed Issue: Agent ignores retrieved text in favor of internal weights.
- Recommended Action: Implement a 'strict grounding' prompt step or an evidence-verification layer before output.
- Expected Improvement: Reduction in factual contradictions.
Next steps
- Review the temperature settings on the generation LLM to reduce creative drift.
- Update the system prompt to explicitly forbid answering if the retrieved text does not contain the specific error code definition.
- Add the 'Error Code 502' case to your regression test pack.
Connects securely to your tools. The creator never sees your data.
About this skill
Developing RAG systems often leads to silent failures like hallucinations, poor retrieval ranking, or ungrounded answers. This skill provides a systematic framework for evaluating how AI agents use supplied knowledge bases to ensure factual accuracy and citation integrity.
What it does
- Knowledge auditing analyzes documents to identify coverage gaps and potential retrieval risks.
- Test set generation creates balanced suites of fact lookups, multi-hop queries, and out-of-scope edge cases.
- Grounding evaluation compares agent responses against retrieved passages to detect unsupported claims or hallucinations.
- Retrieval diagnostics assesses passage relevance, context preservation, and chunking effectiveness.
- Performance scoring applies a consistent 0-5 scale across dimensions like faithfulness, completeness, and abstention behavior.
- Regression planning produces reusable test packs with clear pass/fail criteria for continuous improvement.
How it works
- Scope confirmation defines the evaluation objectives and identifies available evidence like logs or source excerpts.
- Analysis execution inspects retrieved passages and generated answers against the ground-truth knowledge base.
- Failure classification assigns evidence-based labels such as "relevant source not retrieved" or "answer adds unsupported details."
- Report generation delivers a structured audit including scores, failure patterns, and prioritized technical recommendations.
Frameworks & tools
This tool works with any RAG architecture, vector database, or LLM orchestration framework. It requires text-based inputs such as knowledge base excerpts, retrieval logs (JSON/text), or agent interaction histories.
Why this beats prompting it yourself
Generic prompts often fail to distinguish between retrieval failures and generation hallucinations. This skill enforces a rigorous multi-step inspection that separates factual correctness from writing fluency, preventing the "hallucination of accuracy" in fluent but ungrounded agent responses.
Use cases
- Auditing a customer support bot to ensure it only answers from official manuals.
- Debugging why an agent fails to synthesize information across multiple documents.
- Validating citation accuracy in legal or technical documentation assistants.
- Testing RAG abstention logic for queries that fall outside the knowledge base.
Known limitations
Cannot directly connect to live vector databases or production telemetry. Evaluations are limited to the specific excerpts and logs provided in the session.
Cannot connect to live databases or production telemetry. Evaluation is restricted to provided excerpts and logs. Does not tune RAG pipelines or change embeddings directly.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 15 days ago
Permissions
No special permissions declared or detected