Evaluating naive and hybrid RAG using Weaviate and Athina

Originally published on Medium on 2024-11-07. This November 2024 tutorial uses the older Weaviate and LangChain APIs. The examples have not been retested against current SDK releases. Original diagrams and evaluation screenshots are available on Medium.

In the race to integrate AI into enterprise operations, technical leaders face a critical challenge: how to harness the power of Large Language Models while ensuring responses align with organizational truth. Retrieval-Augmented Generation (RAG) emerges as the architectural solution that bridges this gap. By enabling LLMs to dynamically access and leverage your organization’s proprietary knowledge base, RAG transforms generic AI capabilities into precision-engineered, business-specific intelligence. Now there are many RAG techniques being used but in this blog we will particularly learn about Naive RAG and Hybrid RAG which are the most commonly used techniques. Not only we will build a RAG pipeline using a dataset, we will also run context sufficiency and response evaluations using Athina AI so that we have a better idea of what approach suits our organisational data. Let’s dive straight into it.

Differentiating Between Standard and Hybrid RAG Models

Standard RAG Models: These typically involve a straightforward integration of a single transformer-based model with a document retriever. Documents are embedded into vectors, stored in a vector database, and when a query comes in, it’s embedded and used to retrieve similar chunks through vector similarity search. These chunks then augment the LLM’s prompt along with the original query to generate the final answer. Simple, effective, and straightforward — that’s Naive RAG in action! Hybrid RAG Models: Hybrid RAG employs a more sophisticated dual-path architecture where documents undergo both semantic and keyword-based processing, creating two parallel information retrieval systems — one based on dense vector embeddings for semantic understanding, and another using sparse representations for keyword matching. When a query arrives, it’s processed through both paths simultaneously, allowing the system to leverage both semantic similarity and keyword relevance, with the results from both approaches being merged and reranked before being used to augment the LLM’s prompt, ultimately providing more comprehensive and accurate responses by combining the strengths of both retrieval methods.

Why did we choose Weaviate as our Vector DB?

A vector database is a key part of a Retrieval-Augmented Generation (RAG) system, powering efficient, accurate storage and retrieval of high-dimensional data, which boosts the performance of AI-driven search and retrieval. We chose Weaviate as it has the following benefits over other Vector DB’s

1. Performance and Speed: Weaviate has low-latency responses and high query speeds to keep our RAG system responsive and efficient.
2. Scalability: Weaviate can scale horizontally, handling increasing amounts of data without performance dips.
3. Search Flexibility: Weaviate supports both hybrid search (combining vector and keyword search) and normal vector search to ensure highly relevant results.
4. Developer Support: Weaviate has robust SDKs, good documentation, and an active community to streamline development and integration. Now since Weaviate ticks all these boxes, we will use that for the purpose of this blog.

Setting up Weaviate

Setting up weaviate for the very first time can be a little tricky but this blog acts a complete guide on how to do it.

• Simply sign up on the Weaviate
• Create a cluster and open its connection details.
• Copy the Rest Endpoint and the Admin API key (which you will find on same page). Both of these will be used while connecting pur RAG application with Weaviate.

Evaluation Metrics

Evaluating our RAG is one of the major challenges that Gen AI industry is facing today. For tackling with this problem, we will use Athina AI which provides an amazing platform for evaluating our prompts, contexts and our responses. Lets dive straight into it and see how it works.

Setting up Athina

Playing around with Athina is pretty straightforward.

• Simply sign up on the platform

• Upload your dataset (if you already have one). You have the flexibility to upload JSON, CSV, App Logging. Import from Hugging Face etc

• Now after you upload your dataset, you can run evals on your dataset. It ranges from different RAGAS evaluations on context (which we will use today) plus different evals on results.

RAG Implementation

Step 1: Preparation of Dataset

Now since we wanted to compare the results of both Hybrid and Naive RAG, we prepared a dataset with the help of ChatGPT. Here are the steps followed:

• Firstly, we collected 3 books having 18–19 pages in each book
• Then we used ChatGPT to generate questions and ground truths out of those books.
• Now for ground truth, we manually verified the answers from all the books to ensure that our LLM didn’t hallucinated.

Now, we want to run 2 evaluations on our dataset which is:

1. Context Sufficiency: We want to check whether documents/context fetched by both Naive and Hybrid RAG are sufficient to answer the question. So we would need to generate 2 columns for this: one for Naive one and one for Hybrid one. In this blog, I will walk you through on how to generate those columns.
2. Ragas Answer Semantic Similarity: We want to check if the answer produced by these RAG pipelines are upto the mark or not. So we will run a semantic similarity search on results generated by both methods and the ground truth.

Step 2: Implementation of Naive and Hybrid RAG

Firstly we will implement a normal RAG Pipeline using Weaviate as a Vector DB. Now there are some special points we should take care of while implementing this.

• Weaviate uses a concept of classes which we will use and pass a parameter while initializing our Vector DB.
• This class has to be same while implementing both Naive and Hybrid RAG and the class has to use a certain vectorizer which we will see below.

!pip install -q langchain sentence-transformers cohere faiss-cpu rank_bm25 langchain-community openai tiktoken weaviate-client

from langchain.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.retrievers import ContextualCompressionRetriever
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain.retrievers import BM25Retriever, EnsembleRetriever
import os
import pandas as pd
from langchain.docstore.document import Document
from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI
#Reading dataset
documents=[]
df = pd.read_csv("/content/Full_Topic_Dataset_with_Questions.csv",encoding='latin-1')

if 'Description' in df.columns:
    print(df['Description'])
    documents.extend(Document(page_content=df['Description'][i]) for i in range(len(df['Description'])))
#Splitting Texts
text_splitter=RecursiveCharacterTextSplitter(chunk_size=512,chunk_overlap=50)
text_splits=text_splitter.split_documents(documents)
print(len(text_splits))
api_key="key here" # use this or import from env variables
#Defining Embeedings
embeddings = OpenAIEmbeddings(
    openai_api_key= api_key
)
# Initialize GPT-4
llm = ChatOpenAI(
    model_name="gpt-4o",  # or "gpt-4" for the base model
    temperature=0.7,
    openai_api_key= api_key
)

Initializing Weaviate Client and Class

from langchain.retrievers.weaviate_hybrid_search import WeaviateHybridSearchRetriever
import os
import weaviate
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Weaviate
# Create a custom prompt template
prompt_template = """Use the following pieces of context to answer the question at the end.
If you don't know the answer, just say that you don't know, don't try to make up an answer.
Context: {context}
Question: {question}
Answer:"""
PROMPT = PromptTemplate(
    template=prompt_template,
    input_variables=["context", "question"]
)
client = weaviate.Client(
    url= "REST Endpoint from Wevaite Home Page",
    auth_client_secret=weaviate.AuthApiKey(api_key="Wevaite Admin Key here"),
    additional_headers={
        "X-OpenAI-Api-Key": api_key,
    }
)
# Create the schema if it doesn't exist
if not client.schema.exists("AthinaDemo04"):
    class_obj = {
        "class": "AthinaDemo04",
        "properties": [
            {
                "name": "text",
                "dataType": ["text"]
            }
        ],
        "vectorizer": "text2vec-openai"
    }
    client.schema.create_class(class_obj)

Generating Contexts and Answers from Naive RAG

# Create the vector store with correct schema
vectorstore = Weaviate.from_documents(
    documents=text_splits,
    embedding=embeddings,
    client=client,
    index_name="AthinaDemo04",
    by_text=False
)
# Initialize the retriever before creating the chain
retriever_vectordb = vectorstore.as_retriever(search_kwargs={"k": 5})

# Create naive RAG chain
naive_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever_vectordb,
    return_source_documents=True,
    chain_type_kwargs={"prompt": PROMPT}
)
for index, row in df.iterrows():
    question = row['Question']
    result = naive_chain({"query": question})
    retrieved_docs = retriever_vectordb.get_relevant_documents(question)
    doc_contents = []

    # Loop through each retrieved document and collect their contents
    for doc in retrieved_docs:
        doc_contents.append(doc.page_content)

    # Convert the list of document contents to a single string
    doc_contents_str = '; '.join(doc_contents)  # Using a delimiter like semicolon to separate documents
    # Store the string of document contents in a new column associated with the row
    df.at[index, 'Naive RAG Context'] = doc_contents_str
    df.at[index, 'Naive RAG Answer'] = result['result']
# Adding a new column of contexts and inserting into sheet
output_csv="new.csv"
df.to_csv(output_csv, index=False)

Generating Contexts and Answers from Hybrid RAG

# Create the hybrid retriever
hybrid_weaviate_retriever = WeaviateHybridSearchRetriever(
    client=client,
    index_name="AthinaDemo04",  # Make sure this matches your vectorstore index_name
    text_key="text",
    k=4,
    alpha=0.5,
    embedding=embeddings,
    attributes=[]
)
# Create the Hybrid QA chain
hybrid_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=hybrid_weaviate_retriever,
    return_source_documents=True,
    chain_type_kwargs={"prompt": PROMPT}
)
# Process your questions
for index, row in df.iterrows():
    question = row['Question']
    result = hybrid_chain({"query": question})
    retrieved_docs = hybrid_weaviate_retriever.get_relevant_documents(question)
    doc_contents = []

    for doc in retrieved_docs:
        doc_contents.append(doc.page_content)
    doc_contents_str = '; '.join(doc_contents)
    df.at[index, 'Hybrid RAG Context'] = doc_contents_str
    df.at[index, 'Hybrid RAG Answer'] = result['result']
# Save results
df.to_csv("new.csv", index=False)

The resulting CSV contains the question, ground truth, retrieved context, and generated answer for both pipelines.

Evaluating our RAG Application

We will use Athina AI (a Y-Combinator Startup) to evaluate our contexts and results.

1. Context Sufficiency: This evaluation checks if the context contains enough information to answer the user’s query. For checking an LLM call is being done for evaluation and then boolean flags are generated stating whether the context (documents retrieved by retriever) was upto the mark or not. Running this on Athina is fairly simply

• Upload the dataset file (CSV in our case) on Athina IDE
• Use the Evaluate Option on top right and then choose Context Sufficiency
• Now map the columns and choose the LLM Model you want to use (please don’t forget to add the API key in settings)
• and that’s it. Run Evals. Repeat the process for both Naive and Hybrid RAG.

Results: In this experiment, hybrid RAG performed slightly better on context sufficiency. The original evaluation screenshots are linked above.

2. Ragas Answer Semantic Similarity: Now similarly we run semantic similarity evaluation on the responses generated by both Naive and Hybrid RAG. In this evaluation, we compared the ground and response generated individually and generated the results below.

Results: Hybrid search also performed slightly better on answer similarity in this 30-row dataset. That is a small experiment, not evidence that hybrid retrieval always wins. The examples use different retrieval counts (five for naive and four for hybrid), so a more controlled comparison should hold those settings constant and test a larger, representative dataset.

Colab notebook

Please find the Colab notebook attached for replicating the results.

Conclusion

In our comprehensive evaluation of Naive and Hybrid RAG implementations using Weaviate and Athina AI, we’ve demonstrated the distinct advantages and practical applications of both approaches. While Naive RAG offers simplicity and efficiency for straightforward information retrieval tasks, Hybrid RAG’s sophisticated dual-path architecture showed superior contextual understanding by combining semantic similarity with keyword matching. Our experiments with Weaviate as the vector database highlighted its robust performance and seamless integration capabilities. The evaluation metrics provided by Athina AI offered valuable insights into context sufficiency and response quality, enabling organizations to make data-driven decisions about their RAG implementation strategy. As enterprises continue to integrate AI into their operations, the choice between Naive and Hybrid RAG should be guided by specific use cases, data complexity, and required accuracy levels. This blog provides a practical framework for organizations to evaluate and implement RAG solutions that align with their unique knowledge management needs.