Rag Eval

Diagnostic Tools for RAG Performance

Pinpointing why a Retrieval-Augmented Generation (RAG) system is failing can be a guessing game. Is the embedding model weak? Is the chunking strategy off? Or is the LLM simply hallucinating despite having the right context? This skill eliminates the guesswork by providing a standardized evaluation framework for your retrieval pipeline.

Data-Driven Insights

By comparing your retriever's output against a labeled ground-truth set, this tool calculates industry-standard metrics including Recall@k, Precision@k, Hit-Rate, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (nDCG). It goes beyond raw numbers to provide a technical verdict on where your bottleneck lies.

What it helps you solve

• Low Recall: Identifies when your embeddings, chunking strategy, or indexing are failing to surface relevant documents.
• Ranking Issues: Detects when relevant documents are being retrieved but ranked too low for the LLM's context window.
• Generation Bottlenecks: Confirms when retrieval is healthy, indicating that your issues stem from the prompt or the LLM's reasoning capabilities.

This developer-centric tool requires zero heavy dependencies, running on the Python standard library for easy integration into CI/CD pipelines or local development workflows.