Q&A over LangChain Docs#

Let’s evaluate your architecture on a Q&A dataset for the LangChain python docs. For more examples of how to test different embeddings, indexing strategies, and architectures, see the Evaluating RAG Architectures on Benchmark Tasks notebook.

Pre-requisites#

We will install quite a few prerequisites for this example since we are comparing many techniques and models.

We will be using LangSmith to capture the evaluation traces. You can make a free account at smith.langchain.com. Once you’ve done so, you can make an API key and set it below.

%pip install -U --quiet langchain langsmith langchainhub langchain_benchmarks
%pip install --quiet chromadb openai huggingface pandas langchain_experimental sentence_transformers pyarrow anthropic tiktoken

For this code to work, please configure LangSmith environment variables with your credentials.

import os

os.environ["LANGCHAIN_ENDPOINT"] = "https://api.smith.langchain.com"
os.environ["LANGCHAIN_API_KEY"] = "ls_..."  # Your API key

# Update these with your own API keys
os.environ["ANTHROPIC_API_KEY"] = "sk-..."
os.environ["OPENAI_API_KEY"] = "sk-..."
# Silence warnings from HuggingFace
os.environ["TOKENIZERS_PARALLELISM"] = "false"

import uuid

# Generate a unique run ID for this experiment
run_uid = uuid.uuid4().hex[:6]

Review Q&A Tasks#

The registry provides configurations to test out common architectures on curated datasets. You can review retrieval tasks by filtering on the Type.

from langchain_benchmarks import clone_public_dataset, registry

registry = registry.filter(Type="RetrievalTask")
registry

Name	Type	Dataset ID	Description
LangChain Docs Q&A	RetrievalTask	452ccafc-18e1-4314-885b-edd735f17b9d	Questions and answers based on a snapshot of the LangChain python docs. The environment provides the documents and the retriever information. Each example is composed of a question and reference answer. Success is measured based on the accuracy of the answer relative to the reference answer. We also measure the faithfulness of the model's response relative to the retrieved documents (if any).
Semi-structured Reports	RetrievalTask	c47d9617-ab99-4d6e-a6e6-92b8daf85a7d	Questions and answers based on PDFs containing tables and charts. The task provides the raw documents as well as factory methods to easily index them and create a retriever. Each example is composed of a question and reference answer. Success is measured based on the accuracy of the answer relative to the reference answer. We also measure the faithfulness of the model's response relative to the retrieved documents (if any).

langchain_docs = registry["LangChain Docs Q&A"]
langchain_docs

Name	LangChain Docs Q&A
Type	RetrievalTask
Dataset ID	452ccafc-18e1-4314-885b-edd735f17b9d
Description	Questions and answers based on a snapshot of the LangChain python docs. The environment provides the documents and the retriever information. Each example is composed of a question and reference answer. Success is measured based on the accuracy of the answer relative to the reference answer. We also measure the faithfulness of the model's response relative to the retrieved documents (if any).
Retriever Factories	basic, parent-doc, hyde
Architecture Factories	conversational-retrieval-qa
get_docs

Clone the dataset#

Once you’ve selected the LangChain Docs Q&A task, clone the dataset to your LangSmith tenant. This step requires that your LANGCHAIN_API_KEY be set above.

clone_public_dataset(langchain_docs.dataset_id, dataset_name=langchain_docs.name)

Dataset LangChain Docs Q&A already exists. Skipping.
You can access the dataset at https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/3f29798f-5939-4643-bd99-008ca66b72ed.

Create the index#

When creating a retrieval Q&A system, the first step is to prepare the retriever. How you construct the index significantly impact your system’s performance. Before trying anything too tricky, it’s good benchmark a reliable baseline.

In this case, our baseline will be to generate a single vector for each raw source document and store them directly in a vector store.

Below, fetch the source docs from the cache in GCS. This cache was formed using an ingestion script that scraped the LangChain documentation. To save time and to ensure that the dataset answers are still correct, we will use these source docs for all benchmark approaches.

docs = list(langchain_docs.get_docs())
print(repr(docs[0])[:100] + "...")

Document(page_content="LangChain cookbook | 🦜️🔗 Langchain\n\n[Skip to main content](#docusaurus_skip...

Now we will populate our vectorstore. We will use LangChain’s indexing API to cache embeddings

from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores.chroma import Chroma

embeddings = HuggingFaceEmbeddings(
    model_name="thenlper/gte-base",
    # model_kwargs={"device": 0},  # Comment out to use CPU
)

vectorstore = Chroma(
    collection_name="lcbm-b-huggingface-gte-base",
    embedding_function=embeddings,
    persist_directory="./chromadb",
)

vectorstore.add_documents(docs)
retriever = vectorstore.as_retriever(search_kwargs={"k": 6})

Define the response generator#

Halfway done with our RAG system. We’ve made the Retriever. Now time for the response Generator.

from operator import itemgetter
from typing import Sequence

from langchain.chat_models import ChatAnthropic
from langchain.prompts import ChatPromptTemplate
from langchain.schema.document import Document
from langchain.schema.output_parser import StrOutputParser
from langchain.schema.runnable.passthrough import RunnableAssign


# After the retriever fetches documents, this
# function formats them in a string to present for the LLM
def format_docs(docs: Sequence[Document]) -> str:
    formatted_docs = []
    for i, doc in enumerate(docs):
        doc_string = (
            f"<document index='{i}'>\n"
            f"<source>{doc.metadata.get('source')}</source>\n"
            f"<doc_content>{doc.page_content}</doc_content>\n"
            "</document>"
        )
        formatted_docs.append(doc_string)
    formatted_str = "\n".join(formatted_docs)
    return f"<documents>\n{formatted_str}\n</documents>"


prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an AI assistant answering questions about LangChain."
            "\n{context}\n"
            "Respond solely based on the document content.",
        ),
        ("human", "{question}"),
    ]
)
llm = ChatAnthropic(model="claude-2.1", temperature=1)

response_generator = (prompt | llm | StrOutputParser()).with_config(
    run_name="GenerateResponse",
)

# This is the final response chain.
# It fetches the "question" key from the input dict,
# passes it to the retriever, then formats as a string.

chain = (
    RunnableAssign(
        {
            "context": (itemgetter("question") | retriever | format_docs).with_config(
                run_name="FormatDocs"
            )
        }
    )
    # The "RunnableAssign" above returns a dict with keys
    # question (from the original input) and
    # context: the string-formatted docs.
    # This is passed to the response_generator above
    | response_generator
)

chain.invoke({"question": "What's expression language?"})

' The LangChain Expression Language (LCEL) is a declarative way to easily compose chains of different components like prompts, models, parsers, etc. \n\nSome key things it provides:\n\n- Streaming support - Ability to get incremental outputs from chains rather than waiting for full completion. Useful for long-running chains.\n\n- Async support - Chains can be called synchronously (like in a notebook) or asynchronously (like in production). Same code works for both.\n\n- Optimized parallel execution - Steps that can run in parallel (like multiple retrievals) are automatically parallelized to minimize latency.\n\n- Retries and fallbacks - Help make chains more robust to failure.\n\n- Access to intermediate results - Useful for debugging or showing work-in-progress.\n\n- Input and output validation via schemas - Enables catching issues early.\n\n- Tracing - Automatic structured logging of all chain steps for observability.\n\n- Seamless deployment - LCEL chains can be easily deployed with LangServe.\n\nThe key idea is it makes it very easy to take a prototype LLM application made with components like prompts and models and turn it into a robust, scalable production application without changing any code.'

Evaluate#

Let’s evaluate your RAG architecture on the dataset now.

from langsmith.client import Client

from langchain_benchmarks.rag import get_eval_config

client = Client()
RAG_EVALUATION = get_eval_config()

test_run = client.run_on_dataset(
    dataset_name=langchain_docs.name,
    llm_or_chain_factory=chain,
    evaluation=RAG_EVALUATION,
    project_name=f"claude-2 qa-chain simple-index {run_uid}",
    project_metadata={
        "index_method": "basic",
    },
    verbose=True,
)

View the evaluation results for project 'claude-2 qa-chain simple-index 1bdbe5' at:
https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/projects/p/3fe31959-95e8-4413-aa09-620bd49bd0d3?eval=true

View all tests for Dataset LangChain Docs Q&A at:
https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/3f29798f-5939-4643-bd99-008ca66b72ed
[------------------------------------------------->] 86/86
 Eval quantiles:
                                0.25        0.5       0.75       mean  \
embedding_cosine_distance   0.088025   0.115760   0.159969   0.129161   
score_string:accuracy       0.500000   0.700000   1.000000   0.645349   
faithfulness                0.700000   1.000000   1.000000   0.812791   
execution_time             27.098772  27.098772  27.098772  27.098772   

                                mode  
embedding_cosine_distance   0.048622  
score_string:accuracy       0.700000  
faithfulness                1.000000  
execution_time             27.098772  

test_run.get_aggregate_feedback()

Q&A over LangChain Docs

Contents