This project implements a comprehensive Retrieval Augmented Generation (RAG) evaluation pipeline to measure key performance indicators of LLM responses, particularly focusing on hallucination rates and answer faithfulness, using the Ragas framework with ground-truth datasets.

Large Language Models (LLMs) integrated into RAG systems can sometimes generate responses that are plausible but factually incorrect or inconsistent with the provided source context. This phenomenon, known as "hallucination," undermines the reliability and trustworthiness of RAG applications in production environments. This pipeline addresses the critical need for a systematic and automated approach to quantify and monitor hallucination rates, alongside other essential metrics like answer relevancy and context utilization, against ground-truth data. The goal is to ensure the factual accuracy and faithfulness of LLM outputs, enabling developers to build more robust and dependable RAG systems.

+-------------------+      +---------------------+      +---------------------+
|    Data Layer     |      |    RAG Pipeline     |      |   Evaluation Layer  |
|-------------------|      |---------------------|      |---------------------|
| - Ground-truth    |      | - Document Chunking |      | - Ragas Dataset     |
|   Dataset Loader  |<-----|- Embedding Gen.    |<-----|   Conversion      |
|   (CSV/JSON)      |      | - Vector Store      |      | - Run Ragas Metrics |
| - Document Corpus |      |   Indexing          |      |   (Faithfulness,    |
|   Loader          |      | - Retriever         |      |   Answer Relevancy, |
| - Dataset Schema  |      | - LLM Response      |      |   Context Precision,|
|   Validation      |      |   Generation        |      |   Context Recall)   |
+-------------------+      +---------------------+      | - Hallucination     |
         |                                               |   Score (1 - Faith.)|
         |                                               +----------^----------+
         |                                                          |
         +----------------------------------------------------------+
                                     |
                                     v
                           +---------------------+
                           |      Reporting      |
                           |---------------------|
                           | - Aggregate Scores  |
                           | - Global Metrics    |
                           |   Summary           |
                           | - Export Results    |
                           |   (JSON + CSV)      |
                           +---------------------+

Ragas defines Faithfulness as the degree to which the generated answer is grounded in the provided context. A high faithfulness score indicates that the LLM's response can be directly attributed to the retrieved documents, thereby minimizing hallucination.

The hallucination rate in this pipeline is explicitly defined and computed as: hallucination = 1 - faithfulness

This means a faithfulness score of 0.9 would result in a hallucination score of 0.1 (or 10%). Ragas achieves this by:

1. Fact Extraction: It first extracts atomic facts from the generated answer.
2. Context Verification: For each extracted fact, it checks if that fact is supported by the retrieved context.
3. Faithfulness Score: The faithfulness score is the ratio of supported facts to the total number of facts extracted from the answer.

1. Clone the repository:

   git clone https://github.com/your-repo/rag-evaluation-pipeline.git
   cd rag-evaluation-pipeline

2. Create a virtual environment (Python 3.10+ recommended):

   python -m venv venv
   # On Windows
   .\venv\Scripts\activate
   # On macOS/Linux
   source venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt

4. Configure environment variables: Copy the .env.example file to .env and fill in your API keys.

   cp .env.example .env
   # Open .env and add your OpenAI or Azure OpenAI keys

   Example .env content:

   OPENAI_API_KEY="sk-YOUR_OPENAI_API_KEY"
   
   # Or for Azure OpenAI
   # AZURE_OPENAI_API_KEY="your_azure_openai_api_key_here"
   # AZURE_OPENAI_ENDPOINT="https://your-resource-name.openai.azure.com/"
   # AZURE_OPENAI_API_VERSION="2023-05-15"
   # AZURE_OPENAI_DEPLOYMENT_NAME="your-gpt-35-turbo-deployment-name"
   # AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME="your-text-embedding-ada-002-deployment-name"
   
   VECTOR_STORE_PATH="./vectorstore"
   DOCUMENT_PATH="./data/documents"
   GROUND_TRUTH_PATH="./data/ground_truth"
   EVALUATION_OUTPUT_PATH="./evaluation_results"
   
   LLM_MODEL_NAME="gpt-3.5-turbo"
   EMBEDDING_MODEL_NAME="text-embedding-ada-002"
   

The ground-truth dataset should be a CSV or JSON file containing at least the following columns:

• question: The query posed to the RAG system.
• ground_truth_answer: The expected correct answer.
• ground_truth_context: A relevant context snippet that supports the ground_truth_answer.

Example ground_truth.csv:

question,ground_truth_answer,ground_truth_context
"What is the capital of France?","Paris is the capital of France.","Paris is a major European city and the capital of France. It is known for its art, fashion, gastronomy, and culture."
"What is the largest ocean on Earth?","The Pacific Ocean is the largest ocean on Earth.","The Pacific Ocean is the largest and deepest of Earth's five oceanic divisions. It extends from the Arctic in the north to the Southern Ocean in the south."
"Who painted the Mona Lisa?","Leonardo da Vinci painted the Mona Lisa.","The Mona Lisa is a half-length portrait painting by Italian artist Leonardo da Vinci. Considered an archetypal masterpiece of the Italian Renaissance, it has been described as 'the best known, the most visited, the most written about, the most sung about, the most parodied work of art in the world'."

The document corpus can consist of various file types (e.g., .txt, .pdf, .csv, .json). These documents will be chunked and indexed into the vector store.

Example data/documents/france.txt:

France is a country located in Western Europe. Its capital city is Paris. Paris is a major global center for art, fashion, gastronomy, and culture. The Eiffel Tower is a famous landmark in Paris.

To run the entire pipeline, execute the main.py script:

python src/main.py

This script will perform the following steps:

1. Load documents from data/documents/.
2. Chunk the documents and create/load a FAISS vector store in vectorstore/.
3. Initialize the RAG pipeline using the configured LLM and embeddings.
4. Load the ground-truth dataset from data/ground_truth/ground_truth.csv.
5. Generate RAG responses for each question in the ground-truth dataset.
6. Run Ragas evaluation metrics (faithfulness, answer relevancy, context precision, context recall).
7. Calculate the explicit hallucination score (1 - faithfulness).
8. Aggregate and export the detailed and summary results to evaluation_results/.

After running python src/main.py, you will find two files in the evaluation_results/ directory:

evaluation_results_detailed.csv: Contains individual scores for each question.

evaluation_results_aggregated.json: Contains the mean scores for all metrics.

{
    "faithfulness": 0.95,
    "answer_relevancy": 0.98,
    "context_precision": 0.92,
    "context_recall": 0.90,
    "hallucination_score": 0.05
}

(Note: The example output above is illustrative. Actual scores will vary based on the LLM, embedding model, documents, and ground-truth data.)

To incorporate additional Ragas metrics:

1. Import the desired metric from ragas.metrics in src/evaluation/evaluator.py.
2. Add the new metric to the metrics list passed to the evaluate function in src/evaluation/evaluator.py.

1. LLMs:
   • In src/rag/rag_pipeline.py, the get_llm function can be extended to support other langchain_community or langchain_openai LLMs.
   • Ensure any new LLM is compatible with BaseChatForToolCalling or BaseLanguageModel.
   • Update src/config/settings.py to include any new environment variables required for the new LLM.
2. Embedding Models:
   • In src/ingestion/embedding.py, the get_embedding_model function can be extended to support other langchain_community or langchain_openai embedding models.
   • Ensure any new embedding model is compatible with Embeddings.
   • Update src/config/settings.py to include any new environment variables required for the new embedding model.

To use a different vector store (e.g., Chroma instead of FAISS):

1. Modify src/ingestion/vector_store.py to use the new vector store's API (e.g., Chroma.from_documents and Chroma.as_retriever).
2. Update requirements.txt with the necessary package for the new vector store (chromadb).
3. Ensure compatibility with langchain_core.vectorstores.VectorStore.

1. Missing API Keys: The most common issue. Ensure all required API keys (OPENAI_API_KEY, AZURE_OPENAI_API_KEY, etc.) are correctly set in your .env file.
2. Incorrect File Paths: Verify that DOCUMENT_PATH, GROUND_TRUTH_PATH, VECTOR_STORE_PATH, and EVALUATION_OUTPUT_PATH in .env (and settings.py) point to the correct directories relative to the project root.
3. Empty Documents Directory: If data/documents/ is empty, the vector store will not be created, leading to retrieval errors.
4. Ground Truth Format Mismatch: Ensure your ground_truth.csv (or JSON) adheres to the expected column names (question, ground_truth_answer, ground_truth_context).
5. LLM Rate Limits: Large datasets might hit LLM API rate limits. Consider implementing retry mechanisms or reducing the dataset size for initial runs.
6. Out-of-Memory Errors: Processing very large document corpuses or generating responses for extensive ground-truth datasets can consume significant memory. Adjust chunk_size or run on machines with more RAM.
7. Dependency Conflicts: If you encounter ModuleNotFoundError or similar issues, verify that all packages in requirements.txt are installed and that your virtual environment is activated.
8. Azure OpenAI Configuration: For Azure, ensure AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_DEPLOYMENT_NAME, and AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME are correctly configured, matching your Azure OpenAI resource. The AZURE_OPENAI_API_VERSION should also be compatible.
9. Dangerous Deserialization Warning: When loading FAISS vector stores, allow_dangerous_deserialization=True is used. This is acceptable for local, trusted data but should be handled with care in production if the vector store source is untrusted.