Router Retriever

In this guide, we define a custom router retriever that selects one or
more candidate retrievers in order to execute a given query.
The router ("BaseSelector") module uses the LLM to dynamically make
decisions on which underlying retrieval tools to use. This can be
helpful to select one out of a diverse range of data sources. This can
also be helpful to aggregate retrieval results across a variety of
data sources (if a multi-selector module is used).
This notebook is very similar to the RouterQueryEngine notebook.
Setup
   # NOTE: This is ONLY necessary in jupyter notebook.
   # Details: Jupyter runs an event-loop behind the scenes.
   #          This results in nested event-loops when we start an event-loop to make async queries.
   #          This is normally not allowed, we use nest_asyncio to allow it for convenience.
   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().handlers = []
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SummaryIndex,
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
       SimpleKeywordTableIndex,
   )
   from llama_index.llms import OpenAI
   Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   NumExpr defaulting to 8 threads.
Load Data
We first show how to convert a Document into a set of Nodes, and
insert into a DocumentStore.
   # load documents
   documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   # initialize service context (set chunk size)
   llm = OpenAI(model="gpt-4")
   service_context = ServiceContext.from_defaults(chunk_size=1024, llm=llm)
   nodes = service_context.node_parser.get_nodes_from_documents(documents)
   # initialize storage context (by default it's in-memory)
   storage_context = StorageContext.from_defaults()
   storage_context.docstore.add_documents(nodes)
   # define
   summary_index = SummaryIndex(nodes, storage_context=storage_context)
   vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
   keyword_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)
   list_retriever = summary_index.as_retriever()
   vector_retriever = vector_index.as_retriever()
   keyword_retriever = keyword_index.as_retriever()
   from llama_index.tools import RetrieverTool
   list_tool = RetrieverTool.from_defaults(
       retriever=list_retriever,
       description="Will retrieve all context from Paul Graham's essay on What I Worked On. Don't use if the question only requires more specific context.",
   )
   vector_tool = RetrieverTool.from_defaults(
       retriever=vector_retriever,
       description="Useful for retrieving specific context from Paul Graham essay on What I Worked On.",
   )
   keyword_tool = RetrieverTool.from_defaults(
       retriever=keyword_retriever,
       description="Useful for retrieving specific context from Paul Graham essay on What I Worked On (using entities mentioned in query)",
   )
Define Selector Module for Routing
There are several selectors available, each with some distinct
attributes.
The LLM selectors use the LLM to output a JSON that is parsed, and the
corresponding indexes are queried.
The Pydantic selectors (currently only supported by "gpt-4-0613" and
"gpt-3.5-turbo-0613" (the default)) use the OpenAI Function Call API
to produce pydantic selection objects, rather than parsing raw JSON.
Here we use PydanticSingleSelector/PydanticMultiSelector but you can

Router Retriever

use the LLM-equivalents as well.
   from llama_index.selectors.llm_selectors import LLMSingleSelector, LLMMultiSelector
   from llama_index.selectors.pydantic_selectors import (
       PydanticMultiSelector,
       PydanticSingleSelector,
   )
   from llama_index.retrievers import RouterRetriever
   from llama_index.response.notebook_utils import display_source_node
PydanticSingleSelector
   retriever = RouterRetriever(
       selector=PydanticSingleSelector.from_defaults(llm=llm),
       retriever_tools=[
           list_tool,
           vector_tool,
       ],
   )
   # will retrieve all context from the author's life
   nodes = retriever.retrieve(
       "Can you give me all the context regarding the author's life?"
   )
   for node in nodes:
       display_source_node(node)
   Selecting retriever 0: This choice is most relevant as it mentions retrieving all context from the essay, which could include information about the author's life..
**Node ID:** 7d07d325-489e-4157-a745-270e2066a643**Similarity:**
None**Text:** What I Worked On
February 2021
Before college the two main things I worked on, outside of schoo...
**Node ID:** 01f0900b-db83-450b-a088-0473f16882d7**Similarity:**
None**Text:** showed Terry Winograd using SHRDLU. I haven't tried
rereading The Moon is a Harsh Mistress, so I ...
**Node ID:** b2549a68-5fef-4179-b027-620ebfa6e346**Similarity:**
None**Text:** Science is an uneasy alliance between two halves, theory
and systems. The theory people prove thi...
**Node ID:** 4f1e9f0d-9bc6-4169-b3b6-4f169bbfa391**Similarity:**
None**Text:** been explored. But all I wanted was to get out of grad
school, and my rapidly written dissertatio...
**Node ID:** e20c99f9-5e80-4c92-8cc0-03d2a527131e**Similarity:**
None**Text:** stop there, of course, or you get merely photographic
accuracy, and what makes a still life inter...
**Node ID:** dbdf341a-f340-49f9-961f-16b9a51eea2d**Similarity:**
None**Text:** that big, bureaucratic customers are a dangerous source
of money, and that there's not much overl...
**Node ID:** ed341d3a-9dda-49c1-8611-0ab40d04f08a**Similarity:**
None**Text:** about money, because I could sense that Interleaf was on
the way down. Freelance Lisp hacking wor...
**Node ID:** d69e02d3-2732-4567-a360-893c14ae157b**Similarity:**
None**Text:** a web app, is common now, but at the time it wasn't
clear that it was even possible. To find out,...
**Node ID:** df9e00a5-e795-40a1-9a6b-8184d1b1e7c0**Similarity:**
None**Text:** have to integrate with any other software except
Robert's and Trevor's, so it was quite fun to wo...
**Node ID:** 38f2699b-0878-499b-90ee-821cb77e387b**Similarity:**
None**Text:** all too keenly aware of the near-death experiences we

Router Retriever

seemed to have every few months. Nor had I ...
**Node ID:** be04d6a9-1fc7-4209-9df2-9c17a453699a**Similarity:**
None**Text:** for a second still life, painted from the same objects
(which hopefully hadn't rotted yet).
Mean...
**Node ID:** 42344911-8a7c-4e9b-81a8-0fcf40ab7690**Similarity:**
None**Text:** which I'd created years before using Viaweb but had
never used for anything. In one day it got 30...
**Node ID:** 9ec3df49-abf9-47f4-b0c2-16687882742a**Similarity:**
None**Text:** I didn't know but would turn out to like a lot: a woman
called Jessica Livingston. A couple days ...
**Node ID:** d0cf6975-5261-4fb2-aae3-f3230090fb64**Similarity:**
None**Text:** of readers, but professional investors are thinking
"Wow, that means they got all the returns." B...
**Node ID:** 607d0480-7eee-4fb4-965d-3cb585fda62c**Similarity:**
None**Text:** to the "YC GDP," but as YC grows this becomes less and
less of a joke. Now lots of startups get t...
**Node ID:** 730a49c9-55f7-4416-ab91-1d0c96e704c8**Similarity:**
None**Text:** So this set me thinking. It was true that on my current
trajectory, YC would be the last thing I ...
**Node ID:** edbe8c67-e373-42bf-af98-276b559cc08b**Similarity:**
None**Text:** operators you need? The Lisp that John McCarthy
invented, or more accurately discovered, is an an...
**Node ID:** 175a4375-35ec-45a0-a90c-15611505096b**Similarity:**
None**Text:** Like McCarthy's original Lisp, it's a spec rather than
an implementation, although like McCarthy'...
**Node ID:** 0cb367f9-0aac-422b-9243-0eaa7be15090**Similarity:**
None**Text:** must tell readers things they don't already know, and
some people dislike being told such things....
**Node ID:** 67afd4f1-9fa1-4e76-87ac-23b115823e6c**Similarity:**
None**Text:** 1960 paper.
But if so there's no reason to suppose that this is the limit of the
language that m...
   nodes = retriever.retrieve("What did Paul Graham do after RISD?")
   for node in nodes:
       display_source_node(node)
   Selecting retriever 1: The question asks for a specific detail from Paul Graham's essay on 'What I Worked On'. Therefore, the second choice, which is useful for retrieving specific context, is the most relevant..
**Node ID:** 22d20835-7de6-4cf7-92de-2bee339f3157**Similarity:**
0.8017176790752668**Text:** that big, bureaucratic customers are a
dangerous source of money, and that there's not much overl...
**Node ID:** bf818c58-5d5b-4458-acbc-d87cc67a36ca**Similarity:**
0.7935885352785799**Text:** So this set me thinking. It was true that

Router Retriever

on my current trajectory, YC would be the last thing I ...
PydanticMultiSelector
   retriever = RouterRetriever(
       selector=PydanticMultiSelector.from_defaults(llm=llm),
       retriever_tools=[list_tool, vector_tool, keyword_tool],
   )
   nodes = retriever.retrieve(
       "What were noteable events from the authors time at Interleaf and YC?"
   )
   for node in nodes:
       display_source_node(node)
   Selecting retriever 1: This choice is relevant as it allows for retrieving specific context from the essay, which is needed to answer the question about notable events at Interleaf and YC..
   Selecting retriever 2: This choice is also relevant as it allows for retrieving specific context using entities mentioned in the query, which in this case are 'Interleaf' and 'YC'..
   > Starting query: What were noteable events from the authors time at Interleaf and YC?
   query keywords: ['interleaf', 'events', 'noteable', 'yc']
   > Extracted keywords: ['interleaf', 'yc']
**Node ID:** fbdd25ed-1ecb-4528-88da-34f581c30782**Similarity:**
None**Text:** So this set me thinking. It was true that on my current
trajectory, YC would be the last thing I ...
**Node ID:** 4ce91b17-131f-4155-b7b5-8917cdc612b1**Similarity:**
None**Text:** to the "YC GDP," but as YC grows this becomes less and
less of a joke. Now lots of startups get t...
**Node ID:** 9fe6c152-28d4-4006-8a1a-43bb72655438**Similarity:**
None**Text:** stop there, of course, or you get merely photographic
accuracy, and what makes a still life inter...
**Node ID:** d11cd2e2-1dd2-4c3b-863f-246fe3856f49**Similarity:**
None**Text:** of readers, but professional investors are thinking
"Wow, that means they got all the returns." B...
**Node ID:** 2bfbab04-cb71-4641-9bd9-52c75b3a9250**Similarity:**
None**Text:** must tell readers things they don't already know, and
some people dislike being told such things....
   nodes = retriever.retrieve(
       "What were noteable events from the authors time at Interleaf and YC?"
   )
   for node in nodes:
       display_source_node(node)
   Selecting retriever 1: This choice is relevant as it allows for retrieving specific context from the essay, which is needed to answer the question about notable events at Interleaf and YC..
   Selecting retriever 2: This choice is also relevant as it allows for retrieving specific context using entities mentioned in the query, which in this case are 'Interleaf' and 'YC'..
   > Starting query: What were noteable events from the authors time at Interleaf and YC?
   query keywords: ['interleaf', 'yc', 'events', 'noteable']
   > Extracted keywords: ['interleaf', 'yc']
**Node ID:** 49882a2c-bb95-4ff3-9df1-2a40ddaea408**Similarity:**
None**Text:** So this set me thinking. It was true that on my current
trajectory, YC would be the last thing I ...
**Node ID:** d11aced1-e630-4109-8ec8-194e975b9851**Similarity:**
None**Text:** to the "YC GDP," but as YC grows this becomes less and

Router Retriever

less of a joke. Now lots of startups get t...
**Node ID:** 8aa6cc91-8e9c-4470-b6d5-4360ed13fefd**Similarity:**
None**Text:** stop there, of course, or you get merely photographic
accuracy, and what makes a still life inter...
**Node ID:** e37465de-c79a-4714-a402-fbd5f52800a2**Similarity:**
None**Text:** must tell readers things they don't already know, and
some people dislike being told such things....
**Node ID:** e0ac7fb6-84fc-4763-bca6-b68f300ec7b7**Similarity:**
None**Text:** of readers, but professional investors are thinking
"Wow, that means they got all the returns." B...
   nodes = await retriever.aretrieve(
       "What were noteable events from the authors time at Interleaf and YC?"
   )
   for node in nodes:
       display_source_node(node)
   Selecting retriever 1: This choice is relevant as it allows for retrieving specific context from the essay, which is needed to answer the question about notable events at Interleaf and YC..
   Selecting retriever 2: This choice is also relevant as it allows for retrieving specific context using entities mentioned in the query, which in this case are 'Interleaf' and 'YC'..
   > Starting query: What were noteable events from the authors time at Interleaf and YC?
   query keywords: ['events', 'interleaf', 'yc', 'noteable']
   > Extracted keywords: ['interleaf', 'yc']
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=25 request_id=95c73e9360e6473daab85cde93ca4c42 response_code=200
**Node ID:** 76d76348-52fb-49e6-95b8-2f7a3900fa1a**Similarity:**
None**Text:** So this set me thinking. It was true that on my current
trajectory, YC would be the last thing I ...
**Node ID:** 61e1908a-79d2-426b-840e-926df469ac49**Similarity:**
None**Text:** to the "YC GDP," but as YC grows this becomes less and
less of a joke. Now lots of startups get t...
**Node ID:** cac03004-5c02-4145-8e92-c320b1803847**Similarity:**
None**Text:** stop there, of course, or you get merely photographic
accuracy, and what makes a still life inter...
**Node ID:** f0d55e5e-5349-4243-ab01-d9dd7b12cd0a**Similarity:**
None**Text:** of readers, but professional investors are thinking
"Wow, that means they got all the returns." B...
**Node ID:** 1516923c-0dee-4af2-b042-3e1f38de7e86**Similarity:**
None**Text:** must tell readers things they don't already know, and
some people dislike being told such things....


Auto Merging Retriever

In this notebook, we showcase our "AutoMergingRetriever", which looks
at a set of leaf nodes and recursively "merges" subsets of leaf nodes
that reference a parent node beyond a given threshold. This allows us
to consolidate potentially disparate, smaller contexts into a larger
context that might help synthesis.
You can define this hierarchy yourself over a set of documents, or you
can make use of our brand-new text parser: a HierarchicalNodeParser
that takes in a candidate set of documents and outputs an entire
hierarchy of nodes, from "coarse-to-fine".
   %load_ext autoreload
   %autoreload 2
Load Data
Let's first load the Llama 2 paper:
https://arxiv.org/pdf/2307.09288.pdf. This will be our test data.
   !wget --user-agent "Mozilla" "https://arxiv.org/pdf/2307.09288.pdf" -O "data/llama2.pdf"
   from pathlib import Path
   # from llama_hub.file.pdf.base import PDFReader
   from llama_hub.file.pymu_pdf.base import PyMuPDFReader
   loader = PyMuPDFReader()
   # docs0 = loader.load_data(file=Path("./data/llama2.pdf"))
   docs0 = loader.load(file_path=Path("./data/llama2.pdf"))
By default, the PDF reader creates a separate doc for each page. For
the sake of this notebook, we stitch docs together into one doc. This
will help us better highlight auto-merging capabilities that "stitch"
chunks together later on.
   from llama_index import Document
   doc_text = "\n\n".join([d.get_content() for d in docs0])
   docs = [Document(text=doc_text)]
Parse Chunk Hierarchy from Text, Load into Storage
In this section we make use of the "HierarchicalNodeParser". This will
output a hierarchy of nodes, from top-level nodes with bigger chunk
sizes to child nodes with smaller chunk sizes, where each child node
has a parent node with a bigger chunk size.
By default, the hierarchy is:
* 1st level: chunk size 2048
* 2nd level: chunk size 512
* 3rd level: chunk size 128
We then load these nodes into storage. The leaf nodes are indexed and
retrieved via a vector store - these are the nodes that will first be
directly retrieved via similarity search. The other nodes will be
retrieved from a docstore.
   from llama_index.node_parser import HierarchicalNodeParser, SimpleNodeParser
   node_parser = HierarchicalNodeParser.from_defaults()
   nodes = node_parser.get_nodes_from_documents(docs)
   len(nodes)
   1029
Here we import a simple helper function for fetching "leaf" nodes
within a node list. These are nodes that don't have children of their
own.
   from llama_index.node_parser import get_leaf_nodes, get_root_nodes
   leaf_nodes = get_leaf_nodes(nodes)
   len(leaf_nodes)
   795
   root_nodes = get_root_nodes(nodes)
Load into Storage
We define a docstore, which we load all nodes into.
We then define a "VectorStoreIndex" containing just the leaf-level
nodes.
   # define storage context
   from llama_index.storage.docstore import SimpleDocumentStore
   from llama_index.storage import StorageContext
   from llama_index import ServiceContext
   from llama_index.llms import OpenAI
   docstore = SimpleDocumentStore()
   # insert nodes into docstore
   docstore.add_documents(nodes)
   # define storage context (will include vector store by default too)
   storage_context = StorageContext.from_defaults(docstore=docstore)
   service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-3.5-turbo"))

Auto Merging Retriever

   ## Load index into vector index
   from llama_index import VectorStoreIndex
   base_index = VectorStoreIndex(
       leaf_nodes, storage_context=storage_context, service_context=service_context
   )
Define Retriever
   from llama_index.retrievers.auto_merging_retriever import AutoMergingRetriever
   base_retriever = base_index.as_retriever(similarity_top_k=6)
   retriever = AutoMergingRetriever(base_retriever, storage_context, verbose=True)
   # query_str = "What were some lessons learned from red-teaming?"
   # query_str = "Can you tell me about the key concepts for safety finetuning"
   query_str = "What could be the potential outcomes of adjusting the amount of safety data used in the RLHF stage?"
   nodes = retriever.retrieve(query_str)
   base_nodes = base_retriever.retrieve(query_str)
   > Merging 4 nodes into parent node.
   > Parent node id: caf5f81c-842f-46a4-b679-6be584bd6aff.
   > Parent node text: We conduct RLHF by first collecting human preference data for safety similar to Section 3.2.2: an...
   len(nodes)
   3
   len(base_nodes)
   6
   from llama_index.response.notebook_utils import display_source_node
   for node in nodes:
       display_source_node(node, source_length=10000)
**Node ID:** d4d67180-71c8-4328-b3f1-1e98fa42ab69**Similarity:**
0.8694979150607424**Text:** We also list two qualitative examples
where safety and helpfulness reward models don’t agree with each other
in Table 35. A.4.2 Qualitative Results on Safety Data Scaling In
Section 4.2.3, we study the impact of adding more safety data into
model RLHF in a quantitative manner. Here we showcase a few samples to
qualitatively examine the evolution of model behavior when we scale
safety data in Tables 36, 37, and 38. In general, we are observing
that Llama 2-Chat becomes safer responding to unsafe prompts with more
safety data used.
**Node ID:** caf5f81c-842f-46a4-b679-6be584bd6aff**Similarity:**
0.86168727941324**Text:** We conduct RLHF by first collecting human
preference data for safety similar to Section 3.2.2: annotators write
a prompt that they believe can elicit unsafe behavior, and then
compare multiple model responses to the prompts, selecting the
response that is safest according to a set of guidelines. We then use
the human preference data to train a safety reward model (see Section
3.2.2), and also reuse the adversarial prompts to sample from the
model during the RLHF stage. Better Long-Tail Safety Robustness
without Hurting Helpfulness Safety is inherently a long-tail problem,
where the challenge comes from a small number of very specific cases.
We investigate the impact of Safety RLHF by taking two intermediate
Llama 2-Chat checkpoints—one without adversarial prompts in the RLHF
stage and one with them—and score their responses on our test sets
using our safety and helpfulness reward models. In Figure 14, we plot
the score distribution shift of the safety RM on the safety test set
(left) and that of the helpfulness RM on the helpfulness test set
(right). In the left hand side of the figure, we observe that the
distribution of safety RM scores on the safety set shifts to higher
reward scores after safety tuning with RLHF, and that the long tail of

Auto Merging Retriever

the distribution near zero thins out. A clear cluster appears on the
top-left corner suggesting the improvements of model safety. On the
right side, we do not observe any gathering pattern below the y = x
line on the right hand side of Figure 14, which indicates that the
helpfulness score distribution is preserved after safety tuning with
RLHF. Put another way, given sufficient helpfulness training data, the
addition of an additional stage of safety mitigation does not
negatively impact model performance on helpfulness to any notable
degradation. A qualitative example is shown in Table 12. Impact of
Safety Data Scaling. A tension between helpfulness and safety of LLMs
has been observed in previous studies (Bai et al., 2022a). To better
understand how the addition of safety training data affects general
model performance, especially helpfulness, we investigate the trends
in safety data scaling by adjusting the amount of safety data used in
the RLHF stage.
**Node ID:** d9893bef-a5a7-4248-a0a1-d7c28800ae59**Similarity:**
0.8546977459150967**Text:** 0 0.2 0.4 0.6 0.8 1.0 Helpfulness RM Score
before Safety RLHF 0.0 0.2 0.4 0.6 0.8 1.0 Helpfulness RM Score after
Safety RLHF 0 1000 0 1000 Figure 14: Impact of safety RLHF measured by
reward model score distributions. Left: safety reward model scores of
generations on the Meta Safety test set. The clustering of samples in
the top left corner suggests the improvements of model safety.
   for node in base_nodes:
       display_source_node(node, source_length=10000)
**Node ID:** 16328561-9ff7-4307-8d31-adf6bb74b71b**Similarity:**
0.8770715326726375**Text:** A qualitative example is shown in Table
12. Impact of Safety Data Scaling. A tension between helpfulness and
safety of LLMs has been observed in previous studies (Bai et al.,
2022a). To better understand how the addition of safety training data
affects general model performance, especially helpfulness, we
investigate the trends in safety data scaling by adjusting the amount
of safety data used in the RLHF stage.
**Node ID:** e756d327-1a28-4228-ac38-f8a831b1bf77**Similarity:**
0.8728111844788112**Text:** A clear cluster appears on the top-left
corner suggesting the improvements of model safety. On the right side,
we do not observe any gathering pattern below the y = x line on the
right hand side of Figure 14, which indicates that the helpfulness
score distribution is preserved after safety tuning with RLHF. Put
another way, given sufficient helpfulness training data, the addition
of an additional stage of safety mitigation does not negatively impact
model performance on helpfulness to any notable degradation. A
qualitative example is shown in Table 12. Impact of Safety Data
Scaling.
**Node ID:** d4d67180-71c8-4328-b3f1-1e98fa42ab69**Similarity:**
0.8697379697028405**Text:** We also list two qualitative examples
where safety and helpfulness reward models don’t agree with each other
in Table 35. A.4.2 Qualitative Results on Safety Data Scaling In
Section 4.2.3, we study the impact of adding more safety data into
model RLHF in a quantitative manner. Here we showcase a few samples to

Auto Merging Retriever

qualitatively examine the evolution of model behavior when we scale
safety data in Tables 36, 37, and 38. In general, we are observing
that Llama 2-Chat becomes safer responding to unsafe prompts with more
safety data used.
**Node ID:** d9893bef-a5a7-4248-a0a1-d7c28800ae59**Similarity:**
0.855087365309258**Text:** 0 0.2 0.4 0.6 0.8 1.0 Helpfulness RM Score
before Safety RLHF 0.0 0.2 0.4 0.6 0.8 1.0 Helpfulness RM Score after
Safety RLHF 0 1000 0 1000 Figure 14: Impact of safety RLHF measured by
reward model score distributions. Left: safety reward model scores of
generations on the Meta Safety test set. The clustering of samples in
the top left corner suggests the improvements of model safety.
**Node ID:** d62ee107-9841-44b5-8b70-bc6487ad6315**Similarity:**
0.8492541852986794**Text:** Better Long-Tail Safety Robustness without
Hurting Helpfulness Safety is inherently a long-tail problem, where
the challenge comes from a small number of very specific cases. We
investigate the impact of Safety RLHF by taking two intermediate Llama
2-Chat checkpoints—one without adversarial prompts in the RLHF stage
and one with them—and score their responses on our test sets using our
safety and helpfulness reward models.
**Node ID:** 312a63b3-5e28-4fbf-a3e1-4e8dc0c026ea**Similarity:**
0.8488371951811564**Text:** We conduct RLHF by first collecting human
preference data for safety similar to Section 3.2.2: annotators write
a prompt that they believe can elicit unsafe behavior, and then
compare multiple model responses to the prompts, selecting the
response that is safest according to a set of guidelines. We then use
the human preference data to train a safety reward model (see Section
3.2.2), and also reuse the adversarial prompts to sample from the
model during the RLHF stage.
Plug it into Query Engine
   from llama_index.query_engine import RetrieverQueryEngine
   query_engine = RetrieverQueryEngine.from_args(retriever)
   base_query_engine = RetrieverQueryEngine.from_args(base_retriever)
   response = query_engine.query(query_str)
   > Merging 4 nodes into parent node.
   > Parent node id: 3671b20d-ea5e-4afc-983e-02be6ee8302d.
   > Parent node text: We conduct RLHF by first collecting human preference data for safety similar to Section 3.2.2: an...
   print(str(response))
   Adjusting the amount of safety data used in the RLHF stage could potentially have the following outcomes:
   1. Improved model safety: Increasing the amount of safety data used in RLHF may lead to improvements in model safety. This means that the model becomes better at responding to unsafe prompts and avoids generating unsafe or harmful outputs.
   2. Thinning out of the long tail of safety RM scores: Increasing the amount of safety data may result in a shift in the distribution of safety reward model (RM) scores towards higher reward scores. This means that the model becomes more consistent in generating safe responses and reduces the occurrence of low safety scores.
   3. Preservation of helpfulness performance: Adjusting the amount of safety data used in RLHF is not expected to negatively impact model performance on helpfulness. This means that the model's ability to generate helpful responses is maintained even after incorporating additional safety training.

Auto Merging Retriever

   4. Gathering pattern in helpfulness RM scores: There is no observed gathering pattern below the y = x line in the distribution of helpfulness RM scores after safety tuning with RLHF. This suggests that the helpfulness score distribution is preserved, indicating that the model's helpfulness performance is not significantly degraded by the addition of safety mitigation measures.
   Overall, adjusting the amount of safety data used in the RLHF stage aims to strike a balance between improving model safety without compromising its helpfulness performance.
   base_response = base_query_engine.query(query_str)
   print(str(base_response))
   Adjusting the amount of safety data used in the RLHF stage could potentially lead to improvements in model safety. This can be observed by a clear cluster appearing on the top-left corner, suggesting enhanced model safety. Additionally, it is indicated that the helpfulness score distribution is preserved after safety tuning with RLHF, indicating that the addition of safety data does not negatively impact model performance on helpfulness.
Evaluation
We evaluate how well the hierarchical retriever works compared to the
baseline retriever in a more quantitative manner.
**WARNING**: This can be *expensive*, especially with GPT-4. Use
caution and tune the sample size to fit your budget.
   from llama_index.evaluation import (
       DatasetGenerator,
       QueryResponseDataset,
   )
   from llama_index import ServiceContext
   from llama_index.llms import OpenAI
   import nest_asyncio
   nest_asyncio.apply()
   # NOTE: run this if the dataset isn't already saved
   # Note: we only generate from the first 20 nodes, since the rest are references
   eval_service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-4"))
   dataset_generator = DatasetGenerator(
       root_nodes[:20],
       service_context=eval_service_context,
       show_progress=True,
       num_questions_per_chunk=3,
   )
   eval_dataset = await dataset_generator.agenerate_dataset_from_nodes(num=60)
   eval_dataset.save_json("data/llama2_eval_qr_dataset.json")
   # optional
   eval_dataset = QueryResponseDataset.from_json("data/llama2_eval_qr_dataset.json")
Compare Results
We run evaluations on each of the retrievers: correctness, semantic
similarity, relevance, and faithfulness.
   import asyncio
   import nest_asyncio
   nest_asyncio.apply()
   from llama_index.evaluation import (
       CorrectnessEvaluator,
       SemanticSimilarityEvaluator,
       RelevancyEvaluator,
       FaithfulnessEvaluator,
       PairwiseComparisonEvaluator,
   )
   from collections import defaultdict
   import pandas as pd
   # NOTE: can uncomment other evaluators
   evaluator_c = CorrectnessEvaluator(service_context=eval_service_context)
   evaluator_s = SemanticSimilarityEvaluator(service_context=eval_service_context)
   evaluator_r = RelevancyEvaluator(service_context=eval_service_context)
   evaluator_f = FaithfulnessEvaluator(service_context=eval_service_context)
   # pairwise_evaluator = PairwiseComparisonEvaluator(service_context=eval_service_context)
   from llama_index.evaluation.eval_utils import get_responses, get_results_df
   from llama_index.evaluation import BatchEvalRunner
   eval_qs = eval_dataset.questions
   qr_pairs = eval_dataset.qr_pairs
   ref_response_strs = [r for (_, r) in qr_pairs]
   pred_responses = get_responses(eval_qs, query_engine, show_progress=True)
   base_pred_responses = get_responses(eval_qs, base_query_engine, show_progress=True)
   100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:07<00:00,  8.17it/s]
   import numpy as np
   pred_response_strs = [str(p) for p in pred_responses]

Auto Merging Retriever

   base_pred_response_strs = [str(p) for p in base_pred_responses]
   evaluator_dict = {
       "correctness": evaluator_c,
       "faithfulness": evaluator_f,
       "relevancy": evaluator_r,
       "semantic_similarity": evaluator_s,
   }
   batch_runner = BatchEvalRunner(evaluator_dict, workers=2, show_progress=True)
   eval_results = await batch_runner.aevaluate_responses(
       eval_qs, responses=pred_responses, reference=ref_response_strs
   )
   base_eval_results = await batch_runner.aevaluate_responses(
       eval_qs, responses=base_pred_responses, reference=ref_response_strs
   )
   results_df = get_results_df(
       [eval_results, base_eval_results],
       ["Auto Merging Retriever", "Base Retriever"],
       ["correctness", "relevancy", "faithfulness", "semantic_similarity"],
   )
   display(results_df)
                       names  correctness  relevancy  faithfulness  \
   0  Auto Merging Retriever     4.266667   0.916667          0.95
   1          Base Retriever     4.208333   0.916667          0.95
      semantic_similarity
   0             0.962196
   1             0.960602
**Analysis**: The results are roughly the same.
Let's also try to see which answer GPT-4 prefers with our pairwise
evals.
   batch_runner = BatchEvalRunner(
       {"pairwise": pairwise_evaluator}, workers=10, show_progress=True
   )
   pairwise_eval_results = await batch_runner.aevaluate_response_strs(
       eval_qs, response_strs=pred_response_strs, reference=base_pred_response_strs
   )
   pairwise_score = np.array([r.score for r in pairwise_eval_results["pairwise"]]).mean()
   pairwise_score
   0.525
**Analysis**: The pairwise comparison score is a measure of the
percentage of time the candidate answer (using auto-merging retriever)
is preferred vs. the base answer (using the base retriever). Here we
see that it's roughly even.


BM25 Retriever

In this guide, we define a bm25 retriever that search documents using
bm25 method.
This notebook is very similar to the RouterQueryEngine notebook.
Setup
   # NOTE: This is ONLY necessary in jupyter notebook.
   # Details: Jupyter runs an event-loop behind the scenes.
   #          This results in nested event-loops when we start an event-loop to make async queries.
   #          This is normally not allowed, we use nest_asyncio to allow it for convenience.
   import nest_asyncio
   nest_asyncio.apply()
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "sk-..."
   openai.api_key = os.environ["OPENAI_API_KEY"]
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().handlers = []
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
       VectorStoreIndex,
   )
   from llama_index.retrievers import BM25Retriever
   from llama_index.indices.vector_store.retrievers.retriever import VectorIndexRetriever
   from llama_index.llms import OpenAI
Load Data
We first show how to convert a Document into a set of Nodes, and
insert into a DocumentStore.
   # load documents
   documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   # initialize service context (set chunk size)
   llm = OpenAI(model="gpt-4")
   service_context = ServiceContext.from_defaults(chunk_size=1024, llm=llm)
   nodes = service_context.node_parser.get_nodes_from_documents(documents)
   # initialize storage context (by default it's in-memory)
   storage_context = StorageContext.from_defaults()
   storage_context.docstore.add_documents(nodes)
   index = VectorStoreIndex(
       nodes=nodes,
       storage_context=storage_context,
       service_context=service_context,
   )
BM25 Retriever
We will search document with bm25 retriever.
   # !pip install rank_bm25
   # We can pass in the index, doctore, or list of nodes to create the retriever
   retriever = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=2)
   from llama_index.response.notebook_utils import display_source_node
   # will retrieve context from specific companies
   nodes = retriever.retrieve("What happened at Viaweb and Interleaf?")
   for node in nodes:
       display_source_node(node)
**Node ID:** d95537b4-b398-4b47-94ff-da86f05a27f7**Similarity:**
5.171801938898801**Text:** I wanted to go back to RISD, but I was now
broke and RISD was very expensive, so I decided to get...
**Node ID:** 6f84e2a5-1ab1-4389-8799-b7713e085931**Similarity:**
4.838241203957084**Text:** All you had to do was teach SHRDLU more
words.
There weren't any classes in AI at Cornell then, ...
   nodes = retriever.retrieve("What did Paul Graham do after RISD?")
   for node in nodes:
       display_source_node(node)
**Node ID:** a4fd0b29-4138-4741-9e27-9f65d6968eb4**Similarity:**
8.090884087344435**Text:** Not so much because it was badly written as
because the problem is so convoluted. When you're wor...
**Node ID:** d95537b4-b398-4b47-94ff-da86f05a27f7**Similarity:**

BM25 Retriever

5.830874349482576**Text:** I wanted to go back to RISD, but I was now
broke and RISD was very expensive, so I decided to get...
Router Retriever with bm25 method
Now we will combine bm25 retriever with vector index retriever.
   from llama_index.tools import RetrieverTool
   vector_retriever = VectorIndexRetriever(index)
   bm25_retriever = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=2)
   retriever_tools = [
       RetrieverTool.from_defaults(
           retriever=vector_retriever,
           description="Useful in most cases",
       ),
       RetrieverTool.from_defaults(
           retriever=bm25_retriever,
           description="Useful if searching about specific information",
       ),
   ]
   from llama_index.retrievers import RouterRetriever
   retriever = RouterRetriever.from_defaults(
       retriever_tools=retriever_tools,
       service_context=service_context,
       select_multi=True,
   )
   # will retrieve all context from the author's life
   nodes = retriever.retrieve(
       "Can you give me all the context regarding the author's life?"
   )
   for node in nodes:
       display_source_node(node)
   Selecting retriever 0: The author's life context is a broad topic, which may require a comprehensive approach that is useful in most cases..
**Node ID:** fcd399c1-3544-4df3-80a9-0a7d3fd41f1f**Similarity:**
0.7942753162501964**Text:** [10]
Wow, I thought, there's an audience. If I write something and put it
on the web, anyone can...
**Node ID:** b203e140-d549-4284-99f4-b1b5bcd996ea**Similarity:**
0.7788031317604815**Text:** Now all I had to do was learn Italian.
Only stranieri (foreigners) had to take this entrance exa...
Advanced - Hybrid Retriever + Re-Ranking
Here we extend the base retriever class and create a custom retriever
that always uses the vector retriever and BM25 retreiver.
Then, nodes can be re-ranked and filtered. This lets us keep
intermediate top-k values large and letting the re-ranking filter out
un-needed nodes.
To best demonstrate this, we will use a larger set of source documents
-- Chapter 3 from the 2022 IPCC Climate Report.
Setup data
   !curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   100 20.7M  100 20.7M    0     0   361k      0  0:00:58  0:00:58 --:--:--  422k
   # !pip install pypdf
   from llama_index import (
       VectorStoreIndex,
       ServiceContext,
       StorageContext,
       SimpleDirectoryReader,
   )
   from llama_index.llms import OpenAI
   # load documents
   documents = SimpleDirectoryReader(
       input_files=["IPCC_AR6_WGII_Chapter03.pdf"]
   ).load_data()
   # initialize service context (set chunk size)
   # -- here, we set a smaller chunk size, to allow for more effective re-ranking
   llm = OpenAI(model="gpt-3.5-turbo")

BM25 Retriever

   service_context = ServiceContext.from_defaults(chunk_size=256, llm=llm)
   nodes = service_context.node_parser.get_nodes_from_documents(documents)
   # initialize storage context (by default it's in-memory)
   storage_context = StorageContext.from_defaults()
   storage_context.docstore.add_documents(nodes)
   index = VectorStoreIndex(
       nodes, storage_context=storage_context, service_context=service_context
   )
   from llama_index.retrievers import BM25Retriever
   # retireve the top 10 most similar nodes using embeddings
   vector_retriever = index.as_retriever(similarity_top_k=10)
   # retireve the top 10 most similar nodes using bm25
   bm25_retriever = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=10)
Custom Retriever Implementation
   from llama_index.retrievers import BaseRetriever
   class HybridRetriever(BaseRetriever):
       def __init__(self, vector_retriever, bm25_retriever):
           self.vector_retriever = vector_retriever
           self.bm25_retriever = bm25_retriever
       def _retrieve(self, query, **kwargs):
           bm25_nodes = self.bm25_retriever.retrieve(query, **kwargs)
           vector_nodes = self.vector_retriever.retrieve(query, **kwargs)
           # combine the two lists of nodes
           all_nodes = []
           node_ids = set()
           for n in bm25_nodes + vector_nodes:
               if n.node.node_id not in node_ids:
                   all_nodes.append(n)
                   node_ids.add(n.node.node_id)
           return all_nodes
   index.as_retriever(similarity_top_k=5)
   hybrid_retriever = HybridRetriever(vector_retriever, bm25_retriever)
Re-Ranker Setup
   # !pip install sentence_transformers
   from llama_index.indices.postprocessor import SentenceTransformerRerank
   reranker = SentenceTransformerRerank(top_n=4, model="BAAI/bge-reranker-base")
   Downloading (…)lve/main/config.json: 100%|██████████| 799/799 [00:00<00:00, 3.86MB/s]
   Downloading pytorch_model.bin: 100%|██████████| 1.11G/1.11G [00:32<00:00, 34.4MB/s]
   Downloading (…)okenizer_config.json: 100%|██████████| 443/443 [00:00<00:00, 2.19MB/s]
   Downloading (…)tencepiece.bpe.model: 100%|██████████| 5.07M/5.07M [00:00<00:00, 14.1MB/s]
   Downloading (…)cial_tokens_map.json: 100%|██████████| 279/279 [00:00<00:00, 1.48MB/s]
   Use pytorch device: cpu
Retrieve
   from llama_index import QueryBundle
   nodes = hybrid_retriever.retrieve("What is the impact of climate change on the ocean?")
   reranked_nodes = reranker.postprocess_nodes(
       nodes,
       query_bundle=QueryBundle("What is the impact of climate change on the ocean?"),
   )
   print("Initial retrieval: ", len(nodes), " nodes")
   print("Re-ranked retrieval: ", len(reranked_nodes), " nodes")
   Batches: 100%|██████████| 1/1 [00:05<00:00,  5.61s/it]
   Initial retrieval:  19  nodes
   Re-ranked retrieval:  4  nodes

BM25 Retriever

   from llama_index.response.notebook_utils import display_source_node
   for node in reranked_nodes:
       display_source_node(node)
**Node ID:** 74b12b7b-f4b9-490a-9342-b640211468dd**Similarity:**
0.998129665851593**Text:** 3 469Oceans and Coastal Ecosystems and
Their Services  Chapter 3 Frequently Asked Questions FAQ 3...
**Node ID:** 2b35824c-2e96-47b7-8dfb-da25c4eefb7d**Similarity:**
0.996731162071228**Text:** {Box 3.2, 3.2.2.1, 3.4.2.5, 3.4.2.10,
3.4.3.3, Cross-Chapter Box PALEO in Chapter 1} Climate imp...
**Node ID:** 01ef2a9e-0dd0-4bce-ab60-e6a3f6456f7b**Similarity:**
0.9954373240470886**Text:** These ecosystems are also influenced by
non-climate drivers, especially fisheries, oil and gas ex...
**Node ID:** 8a23b728-0352-4b01-a5c0-42765669855d**Similarity:**
0.9872682690620422**Text:** Additionally, climate-change-driven oxygen
loss (Section  3.2.3.2; Luna et  al., 2012; Belley et...
Full Query Engine
   from llama_index.query_engine import RetrieverQueryEngine
   query_engine = RetrieverQueryEngine.from_args(
       retriever=hybrid_retriever,
       node_postprocessors=[reranker],
       service_context=service_context,
   )
   response = query_engine.query("What is the impact of climate change on the ocean?")
   Batches: 100%|██████████| 1/1 [00:05<00:00,  5.74s/it]
   from llama_index.response.notebook_utils import display_response
   display_response(response)
**"Final Response:"** Climate change has significant impacts on the
ocean. It is degrading ocean health and altering stocks of marine
resources. This, combined with over-harvesting, is threatening the
sustenance provided to Indigenous Peoples, the livelihoods of
artisanal fisheries, and marine-based industries such as tourism,
shipping, and transportation. Climate change can also influence human
activities and employment by altering resource availability, spreading
pathogens, flooding shorelines, and degrading ocean ecosystems.
Additionally, increases in intensity, reoccurrence, and duration of
marine heatwaves due to climate change can lead to species
extirpation, habitat collapse, and surpassing ecological tipping
points. Some habitat-forming coastal ecosystems, including coral
reefs, kelp forests, and seagrass meadows, are at high risk of
irreversible phase shifts due to marine heatwaves. Non-climate drivers
such as fisheries, oil and gas extraction, cable laying, and mineral
resource exploration also influence ocean ecosystems.


Ensemble Query Engine Guide

Oftentimes when building a RAG applications there are many retreival
parameters/strategies to decide from (from chunk size to vector vs.
keyword vs. hybrid search, for instance).
Thought: what if we could try a bunch of strategies at once, and have
any AI/reranker/LLM prune the results?
This achieves two purposes:
* Better (albeit more costly) retrieved results by pooling results
  from multiple strategies, assuming the reranker is good
* A way to benchmark different retrieval strategies against each other
  (w.r.t reranker)
This guide showcases this over the Llama 2 paper. We do ensemble
retrieval over different chunk sizes and also different indices.
**NOTE**: A closely related guide is our Ensemble Retrievers Guide -
make sure to check it out!
   %load_ext autoreload
   %autoreload 2
Setup
Here we define the necessary imports.
   # NOTE: This is ONLY necessary in jupyter notebook.
   # Details: Jupyter runs an event-loop behind the scenes.
   #          This results in nested event-loops when we start an event-loop to make async queries.
   #          This is normally not allowed, we use nest_asyncio to allow it for convenience.
   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().handlers = []
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SummaryIndex,
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
   )
   from llama_index.response.notebook_utils import display_response
   from llama_index.llms import OpenAI
   Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   NumExpr defaulting to 8 threads.
Load Data
In this section we first load in the Llama 2 paper as a single
document. We then chunk it multiple times, according to different
chunk sizes. We build a separate vector index corresponding to each
chunk size.
   !wget --user-agent "Mozilla" "https://arxiv.org/pdf/2307.09288.pdf" -O "data/llama2.pdf"
   --2023-09-28 12:56:38--  https://arxiv.org/pdf/2307.09288.pdf
   Resolving arxiv.org (arxiv.org)... 128.84.21.199
   Connecting to arxiv.org (arxiv.org)|128.84.21.199|:443... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 13661300 (13M) [application/pdf]
   Saving to: ‘data/llama2.pdf’
   data/llama2.pdf     100%[===================>]  13.03M   521KB/s    in 42s
   2023-09-28 12:57:20 (320 KB/s) - ‘data/llama2.pdf’ saved [13661300/13661300]
   from pathlib import Path
   from llama_index import Document
   from llama_hub.file.pymu_pdf.base import PyMuPDFReader
   loader = PyMuPDFReader()
   docs0 = loader.load(file_path=Path("./data/llama2.pdf"))
   doc_text = "\n\n".join([d.get_content() for d in docs0])
   docs = [Document(text=doc_text)]
Here we try out different chunk sizes: 128, 256, 512, and 1024.
   # initialize service context (set chunk size)
   llm = OpenAI(model="gpt-4")
   chunk_sizes = [128, 256, 512, 1024]

Ensemble Query Engine Guide

   service_contexts = []
   nodes_list = []
   vector_indices = []
   query_engines = []
   for chunk_size in chunk_sizes:
       print(f"Chunk Size: {chunk_size}")
       service_context = ServiceContext.from_defaults(chunk_size=chunk_size, llm=llm)
       service_contexts.append(service_context)
       nodes = service_context.node_parser.get_nodes_from_documents(docs)
       # add chunk size to nodes to track later
       for node in nodes:
           node.metadata["chunk_size"] = chunk_size
           node.excluded_embed_metadata_keys = ["chunk_size"]
           node.excluded_llm_metadata_keys = ["chunk_size"]
       nodes_list.append(nodes)
       # build vector index
       vector_index = VectorStoreIndex(nodes)
       vector_indices.append(vector_index)
       # query engines
       query_engines.append(vector_index.as_query_engine())
   Chunk Size: 128
   Chunk Size: 256
   Chunk Size: 512
   Chunk Size: 1024
Define Ensemble Retriever
We setup an "ensemble" retriever primarily using our recursive
retrieval abstraction. This works like the following:
* Define a separate "IndexNode" corresponding to the vector retriever
  for each chunk size (retriever for chunk size 128, retriever for
  chunk size 256, and more)
* Put all IndexNodes into a single "SummaryIndex" - when the
  corresponding retriever is called, *all* nodes are returned.
* Define a Recursive Retriever, with the root node being the summary
  index retriever. This will first fetch all nodes from the summary
  index retriever, and then recursively call the vector retriever for
  each chunk size.
* Rerank the final results.
The end result is that all vector retrievers are called when a query
is run.
   # try ensemble retrieval
   from llama_index.tools import RetrieverTool
   from llama_index.schema import IndexNode
   # retriever_tools = []
   retriever_dict = {}
   retriever_nodes = []
   for chunk_size, vector_index in zip(chunk_sizes, vector_indices):
       node_id = f"chunk_{chunk_size}"
       node = IndexNode(
           text=f"Retrieves relevant context from the Llama 2 paper (chunk size {chunk_size})",
           index_id=node_id,
       )
       retriever_nodes.append(node)
       retriever_dict[node_id] = vector_index.as_retriever()
Define recursive retriever.
   from llama_index.selectors.pydantic_selectors import PydanticMultiSelector
   # from llama_index.retrievers import RouterRetriever
   from llama_index.retrievers import RecursiveRetriever
   from llama_index import SummaryIndex
   # the derived retriever will just retrieve all nodes
   summary_index = SummaryIndex(retriever_nodes)
   retriever = RecursiveRetriever(
       root_id="root",
       retriever_dict={"root": summary_index.as_retriever(), **retriever_dict},
   )
Let's test the retriever on a sample query.
   nodes = await retriever.aretrieve(
       "Tell me about the main aspects of safety fine-tuning"
   )
   print(f"Number of nodes: {len(nodes)}")
   for node in nodes:
       print(node.node.metadata["chunk_size"])
       print(node.node.get_text())
Define reranker to process the final retrieved set of nodes.
   # define reranker
   from llama_index.indices.postprocessor import (
       LLMRerank,
       SentenceTransformerRerank,
       CohereRerank,
   )
   # reranker = LLMRerank()
   # reranker = SentenceTransformerRerank(top_n=10)
   reranker = CohereRerank(top_n=10)
Define retriever query engine to integrate the recursive retriever +

Ensemble Query Engine Guide

reranker together.
   # define RetrieverQueryEngine
   from llama_index.query_engine import RetrieverQueryEngine
   query_engine = RetrieverQueryEngine(retriever, node_postprocessors=[reranker])
   response = query_engine.query("Tell me about the main aspects of safety fine-tuning")
   display_response(
       response, show_source=True, source_length=500, show_source_metadata=True
   )
Analyzing the Relative Importance of each Chunk
One interesting property of ensemble-based retrieval is that through
reranking, we can actually use the ordering of chunks in the final
retrieved set to determine the importance of each chunk size. For
instance, if certain chunk sizes are always ranked near the top, then
those are probably more relevant to the query.
   # compute the average precision for each chunk size based on positioning in combined ranking
   from collections import defaultdict
   import pandas as pd
   def mrr_all(metadata_values, metadata_key, source_nodes):
       # source nodes is a ranked list
       # go through each value, find out positioning in source_nodes
       value_to_mrr_dict = {}
       for metadata_value in metadata_values:
           mrr = 0
           for idx, source_node in enumerate(source_nodes):
               if source_node.node.metadata[metadata_key] == metadata_value:
                   mrr = 1 / (idx + 1)
                   break
               else:
                   continue
           # normalize AP, set in dict
           value_to_mrr_dict[metadata_value] = mrr
       df = pd.DataFrame(value_to_mrr_dict, index=["MRR"])
       df.style.set_caption("Mean Reciprocal Rank")
       return df
   # Compute the Mean Reciprocal Rank for each chunk size (higher is better)
   # we can see that chunk size of 256 has the highest ranked results.
   print("Mean Reciprocal Rank for each Chunk Size")
   mrr_all(chunk_sizes, "chunk_size", response.source_nodes)
   Mean Reciprocal Rank for each Chunk Size
            128   256   512   1024
   MRR  0.333333   1.0   0.5  0.25
Evaluation
We more rigorously evaluate how well an ensemble retriever works
compared to the "baseline" retriever.
We define/load an eval benchmark dataset and then run different
evaluations over it.
**WARNING**: This can be *expensive*, especially with GPT-4. Use
caution and tune the sample size to fit your budget.
   from llama_index.evaluation import (
       DatasetGenerator,
       QueryResponseDataset,
   )
   from llama_index import ServiceContext
   from llama_index.llms import OpenAI
   import nest_asyncio
   nest_asyncio.apply()
   # NOTE: run this if the dataset isn't already saved
   eval_service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-4"))
   # generate questions from the largest chunks (1024)
   dataset_generator = DatasetGenerator(
       nodes_list[-1],
       service_context=eval_service_context,
       show_progress=True,
       num_questions_per_chunk=2,
   )
   eval_dataset = await dataset_generator.agenerate_dataset_from_nodes(num=60)
   eval_dataset.save_json("data/llama2_eval_qr_dataset.json")
   # optional
   eval_dataset = QueryResponseDataset.from_json("data/llama2_eval_qr_dataset.json")
Compare Results
   import asyncio
   import nest_asyncio
   nest_asyncio.apply()
   from llama_index.evaluation import (
       CorrectnessEvaluator,
       SemanticSimilarityEvaluator,
       RelevancyEvaluator,
       FaithfulnessEvaluator,
       PairwiseComparisonEvaluator,
   )
   # NOTE: can uncomment other evaluators
   evaluator_c = CorrectnessEvaluator(service_context=eval_service_context)

Ensemble Query Engine Guide

   evaluator_s = SemanticSimilarityEvaluator(service_context=eval_service_context)
   evaluator_r = RelevancyEvaluator(service_context=eval_service_context)
   evaluator_f = FaithfulnessEvaluator(service_context=eval_service_context)
   pairwise_evaluator = PairwiseComparisonEvaluator(service_context=eval_service_context)
   from llama_index.evaluation.eval_utils import get_responses, get_results_df
   from llama_index.evaluation import BatchEvalRunner
   max_samples = 60
   eval_qs = eval_dataset.questions
   qr_pairs = eval_dataset.qr_pairs
   ref_response_strs = [r for (_, r) in qr_pairs]
   # resetup base query engine and ensemble query engine
   # base query engine
   base_query_engine = vector_indices[-1].as_query_engine(similarity_top_k=2)
   # ensemble query engine
   reranker = CohereRerank(top_n=4)
   query_engine = RetrieverQueryEngine(retriever, node_postprocessors=[reranker])
   base_pred_responses = get_responses(
       eval_qs[:max_samples], base_query_engine, show_progress=True
   )
   pred_responses = get_responses(eval_qs[:max_samples], query_engine, show_progress=True)
   import numpy as np
   pred_response_strs = [str(p) for p in pred_responses]
   base_pred_response_strs = [str(p) for p in base_pred_responses]
   evaluator_dict = {
       "correctness": evaluator_c,
       "faithfulness": evaluator_f,
       # "relevancy": evaluator_r,
       "semantic_similarity": evaluator_s,
   }
   batch_runner = BatchEvalRunner(evaluator_dict, workers=1, show_progress=True)
   eval_results = await batch_runner.aevaluate_responses(
       queries=eval_qs[:max_samples],
       responses=pred_responses[:max_samples],
       reference=ref_response_strs[:max_samples],
   )
   base_eval_results = await batch_runner.aevaluate_responses(
       queries=eval_qs[:max_samples],
       responses=base_pred_responses[:max_samples],
       reference=ref_response_strs[:max_samples],
   )
   results_df = get_results_df(
       [eval_results, base_eval_results],
       ["Ensemble Retriever", "Base Retriever"],
       ["correctness", "faithfulness", "semantic_similarity"],
   )
   display(results_df)
                   names  correctness  faithfulness  semantic_similarity
   0  Ensemble Retriever     4.375000      0.983333             0.964546
   1      Base Retriever     4.066667      0.983333             0.956692
   batch_runner = BatchEvalRunner(
       {"pairwise": pairwise_evaluator}, workers=3, show_progress=True
   )
   pairwise_eval_results = await batch_runner.aevaluate_response_strs(
       queries=eval_qs[:max_samples],
       response_strs=pred_response_strs[:max_samples],
       reference=base_pred_response_strs[:max_samples],
   )
   results_df = get_results_df(
       [eval_results, base_eval_results],
       ["Ensemble Retriever", "Base Retriever"],
       ["pairwise"],
   )
   display(results_df)
                    names  pairwise
   0  Pairwise Comparison       0.5


Recursive Retriever + Node References + Braintrust

This guide shows how you can use recursive retrieval to traverse node
relationships and fetch nodes based on "references".
Node references are a powerful concept. When you first perform
retrieval, you may want to retrieve the reference as opposed to the
raw text. You can have multiple references point to the same node.
In this guide we explore some different usages of node references:
* **Chunk references**: Different chunk sizes referring to a bigger
  chunk
* **Metadata references**: Summaries + Generated Questions referring
  to a bigger chunk
We evaluate how well our recursive retrieval + node reference methods
work using Braintrust. Braintrust is the enterprise-grade stack for
building AI products. From evaluations, to prompt playground, to data
management, we take uncertainty and tedium out of incorporating AI
into your business.
You can see example evaluation dashboards here for the:
* base retriever
* recursive metadata retreiver
* recursive chunk retriever
   %load_ext autoreload
   %autoreload 2
   # NOTE: Replace YOUR_OPENAI_API_KEY with your OpenAI API Key and YOUR_BRAINTRUST_API_KEY with your BrainTrust API key. Do not put it in quotes.
   # Signup for Braintrust at https://braintrustdata.com/ and get your API key at https://www.braintrustdata.com/app/braintrustdata.com/settings/api-keys
   # NOTE: Replace YOUR_OPENAI_KEY with your OpenAI API Key and YOUR_BRAINTRUST_API_KEY with your BrainTrust API key. Do not put it in quotes.
   %env OPENAI_API_KEY=
   %env BRAINTRUST_API_KEY=
   %env TOKENIZERS_PARALLELISM=true # This is needed to avoid a warning message from Chroma
   %pip install -U llama_hub llama_index braintrust autoevals pypdf pillow transformers torch torchvision
Load Data + Setup
In this section we download the Llama 2 paper and create an initial
set of nodes (chunk size 1024).
   !mkdir data
   !wget --user-agent "Mozilla" "https://arxiv.org/pdf/2307.09288.pdf" -O "data/llama2.pdf"
   from pathlib import Path
   from llama_hub.file.pdf.base import PDFReader
   from llama_index.response.notebook_utils import display_source_node
   from llama_index.retrievers import RecursiveRetriever
   from llama_index.query_engine import RetrieverQueryEngine
   from llama_index import VectorStoreIndex, ServiceContext
   from llama_index.llms import OpenAI
   import json
   loader = PDFReader()
   docs0 = loader.load_data(file=Path("./data/llama2.pdf"))
   from llama_index import Document
   doc_text = "\n\n".join([d.get_content() for d in docs0])
   docs = [Document(text=doc_text)]
   from llama_index.node_parser import SimpleNodeParser
   from llama_index.schema import IndexNode
   node_parser = SimpleNodeParser.from_defaults(chunk_size=1024)
   base_nodes = node_parser.get_nodes_from_documents(docs)
   # set node ids to be a constant
   for idx, node in enumerate(base_nodes):
       node.id_ = f"node-{idx}"
   from llama_index.embeddings import resolve_embed_model
   embed_model = resolve_embed_model("local:BAAI/bge-small-en")
   llm = OpenAI(model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)
Baseline Retriever
Define a baseline retriever that simply fetches the top-k raw text
nodes by embedding similarity.
   base_index = VectorStoreIndex(base_nodes, service_context=service_context)
   base_retriever = base_index.as_retriever(similarity_top_k=2)

   retrievals = base_retriever.retrieve(
       "Can you tell me about the key concepts for safety finetuning"
   )
   for n in retrievals:
       display_source_node(n, source_length=1500)
   query_engine_base = RetrieverQueryEngine.from_args(
       base_retriever, service_context=service_context
   )
   response = query_engine_base.query(
       "Can you tell me about the key concepts for safety finetuning"
   )
   print(str(response))
Chunk References: Smaller Child Chunks Referring to Bigger Parent Chunk
In this usage example, we show how to build a graph of smaller chunks
pointing to bigger parent chunks.
During query-time, we retrieve smaller chunks, but we follow
references to bigger chunks. This allows us to have more context for
synthesis.
   sub_chunk_sizes = [128, 256, 512]
   sub_node_parsers = [
       SimpleNodeParser.from_defaults(chunk_size=c) for c in sub_chunk_sizes
   ]
   all_nodes = []
   for base_node in base_nodes:
       for n in sub_node_parsers:
           sub_nodes = n.get_nodes_from_documents([base_node])
           sub_inodes = [
               IndexNode.from_text_node(sn, base_node.node_id) for sn in sub_nodes
           ]
           all_nodes.extend(sub_inodes)
       # also add original node to node
       original_node = IndexNode.from_text_node(base_node, base_node.node_id)
       all_nodes.append(original_node)
   all_nodes_dict = {n.node_id: n for n in all_nodes}
   vector_index_chunk = VectorStoreIndex(all_nodes, service_context=service_context)
   vector_retriever_chunk = vector_index_chunk.as_retriever(similarity_top_k=2)
   retriever_chunk = RecursiveRetriever(
       "vector",
       retriever_dict={"vector": vector_retriever_chunk},
       node_dict=all_nodes_dict,
       verbose=True,
   )
   nodes = retriever_chunk.retrieve(
       "Can you tell me about the key concepts for safety finetuning"
   )
   for node in nodes:
       display_source_node(node, source_length=2000)
   query_engine_chunk = RetrieverQueryEngine.from_args(
       retriever_chunk, service_context=service_context
   )
   response = query_engine_chunk.query(
       "Can you tell me about the key concepts for safety finetuning"
   )
   print(str(response))
Metadata References: Summaries + Generated Questions referring to a bigger chunk
In this usage example, we show how to define additional context that
references the source node.
This additional context includes summaries as well as generated
questions.
During query-time, we retrieve smaller chunks, but we follow
references to bigger chunks. This allows us to have more context for
synthesis.
   from llama_index.node_parser import SimpleNodeParser
   from llama_index.schema import IndexNode
   from llama_index.node_parser.extractors import (
       SummaryExtractor,
       QuestionsAnsweredExtractor,
       MetadataExtractor,
   )
   metadata_extractor = MetadataExtractor(
       extractors=[
           SummaryExtractor(summaries=["self"], show_progress=True),
           QuestionsAnsweredExtractor(questions=5, show_progress=True),
       ],
   )
   # run metadata extractor across base nodes, get back dictionaries
   metadata_dicts = metadata_extractor.extract(base_nodes)
   # cache metadata dicts
   def save_metadata_dicts(path):
       with open(path, "w") as fp:
           for m in metadata_dicts:
               fp.write(json.dumps(m) + "\n")
   def load_metadata_dicts(path):
       with open(path, "r") as fp:
           metadata_dicts = [json.loads(l) for l in fp.readlines()]
           return metadata_dicts
   save_metadata_dicts("data/llama2_metadata_dicts.jsonl")
   metadata_dicts = load_metadata_dicts("data/llama2_metadata_dicts.jsonl")

   # all nodes consists of source nodes, along with metadata
   import copy
   all_nodes = copy.deepcopy(base_nodes)
   for idx, d in enumerate(metadata_dicts):
       inode_q = IndexNode(
           text=d["questions_this_excerpt_can_answer"], index_id=base_nodes[idx].node_id
       )
       inode_s = IndexNode(text=d["section_summary"], index_id=base_nodes[idx].node_id)
       all_nodes.extend([inode_q, inode_s])
   all_nodes_dict = {n.node_id: n for n in all_nodes}
   ## Load index into vector index
   from llama_index import VectorStoreIndex, ServiceContext
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(llm=llm)
   vector_index_metadata = VectorStoreIndex(all_nodes, service_context=service_context)
   vector_retriever_metadata = vector_index_metadata.as_retriever(similarity_top_k=2)
   retriever_metadata = RecursiveRetriever(
       "vector",
       retriever_dict={"vector": vector_retriever_metadata},
       node_dict=all_nodes_dict,
       verbose=True,
   )
   nodes = retriever_metadata.retrieve(
       "Can you tell me about the key concepts for safety finetuning"
   )
   for node in nodes:
       display_source_node(node, source_length=2000)
   query_engine_metadata = RetrieverQueryEngine.from_args(
       retriever_metadata, service_context=service_context
   )
   response = query_engine_metadata.query(
       "Can you tell me about the key concepts for safety finetuning"
   )
   print(str(response))
Evaluation
We evaluate how well our recursive retrieval + node reference methods
work using Braintrust. Braintrust is the enterprise-grade stack for
building AI products. From evaluations, to prompt playground, to data
management, we take uncertainty and tedium out of incorporating AI
into your business.
We evaluate both chunk references as well as metadata references. We
use embedding similarity lookup to retrieve the reference nodes. We
compare both methods against a baseline retriever where we fetch the
raw nodes directly. In terms of metrics, we evaluate using both hit-
rate and MRR.
You can see example evaluation dashboards here for the:
* base retriever
* recursive metadata retreiver
* recursive chunk retriever
Dataset Generation
We first generate a dataset of questions from the set of text chunks.
   from llama_index.evaluation import (
       generate_question_context_pairs,
       EmbeddingQAFinetuneDataset,
   )
   import nest_asyncio
   nest_asyncio.apply()
   eval_dataset = generate_question_context_pairs(base_nodes)
   eval_dataset.save_json("data/llama2_eval_dataset.json")
   # optional
   eval_dataset = EmbeddingQAFinetuneDataset.from_json("data/llama2_eval_dataset.json")
Compare Results
We run evaluations on each of the retrievers to measure hit rate and
MRR.
We find that retrievers with node references (either chunk or
metadata) tend to perform better than retrieving the raw chunks.
   import pandas as pd
   # set vector retriever similarity top k to higher
   top_k = 10
   def display_results(names, results_arr):
       """Display results from evaluate."""
       hit_rates = []
       mrrs = []
       for name, eval_results in zip(names, results_arr):
           metric_dicts = []
           for eval_result in eval_results:
               metric_dict = eval_result.metric_vals_dict
               metric_dicts.append(metric_dict)
           results_df = pd.DataFrame(metric_dicts)
           hit_rate = results_df["hit_rate"].mean()
           mrr = results_df["mrr"].mean()
           hit_rates.append(hit_rate)
           mrrs.append(mrr)
       final_df = pd.DataFrame({"retrievers": names, "hit_rate": hit_rates, "mrr": mrrs})

       display(final_df)
Let's define some scoring functions and define our dataset data
variable.
   queries = eval_dataset.queries
   relevant_docs = eval_dataset.relevant_docs
   data = [
       ({"input": queries[query], "expected": relevant_docs[query]})
       for query in queries.keys()
   ]
   def hitRateScorer(input, expected, output=None):
       is_hit = any([id in expected for id in output])
       return 1 if is_hit else 0
   def mrrScorer(input, expected, output=None):
       for i, id in enumerate(output):
           if id in expected:
               return 1 / (i + 1)
       return 0
   import braintrust
   # Evaluate the chunk retriever
   vector_retriever_chunk = vector_index_chunk.as_retriever(similarity_top_k=10)
   retriever_chunk = RecursiveRetriever(
       "vector",
       retriever_dict={"vector": vector_retriever_chunk},
       node_dict=all_nodes_dict,
       verbose=False,
   )
   def runChunkRetriever(input, hooks):
       retrieved_nodes = retriever_chunk.retrieve(input)
       retrieved_ids = [node.node.node_id for node in retrieved_nodes]
       return retrieved_ids
   chunkEval = await braintrust.Eval(
       name="llamaindex-recurisve-retrievers",
       data=data,
       task=runChunkRetriever,
       scores=[hitRateScorer, mrrScorer],
   )
   # Evaluate the metadata retriever
   vector_retriever_metadata = vector_index_metadata.as_retriever(similarity_top_k=10)
   retriever_metadata = RecursiveRetriever(
       "vector",
       retriever_dict={"vector": vector_retriever_metadata},
       node_dict=all_nodes_dict,
       verbose=False,
   )
   def runMetaDataRetriever(input, hooks):
       retrieved_nodes = retriever_metadata.retrieve(input)
       retrieved_ids = [node.node.node_id for node in retrieved_nodes]
       return retrieved_ids
   metadataEval = await braintrust.Eval(
       name="llamaindex-recurisve-retrievers",
       data=data,
       task=runMetaDataRetriever,
       scores=[hitRateScorer, mrrScorer],
   )
   # Evaluate the base retriever
   base_retriever = base_index.as_retriever(similarity_top_k=10)
   def runBaseRetriever(input, hooks):
       retrieved_nodes = base_retriever.retrieve(input)
       retrieved_ids = [node.node.node_id for node in retrieved_nodes]
       return retrieved_ids
   baseEval = await braintrust.Eval(
       name="llamaindex-recurisve-retrievers",
       data=data,
       task=runBaseRetriever,
       scores=[hitRateScorer, mrrScorer],
   )


OnDemandLoaderTool Tutorial

Our "OnDemandLoaderTool" is a powerful agent tool that allows for "on-
demand" data querying from any data source on LlamaHub.
This tool takes in a "BaseReader" data loader, and when called will 1)
load data, 2) index data, and 3) query the data.
In this walkthrough, we show how to use the "OnDemandLoaderTool" to
convert our Wikipedia data loader into an accessible search tool for a
LangChain agent.
   from llama_index.tools.ondemand_loader_tool import OnDemandLoaderTool
   from llama_index.readers.wikipedia import WikipediaReader
   from typing import List
   from pydantic import BaseModel
Define Tool
We first define the "WikipediaReader". Note that the "load_data"
interface to "WikipediaReader" takes in a list of "pages". By default,
this queries the Wikipedia search endpoint which will autosuggest the
relevant pages.
We then wrap it into our "OnDemandLoaderTool".
By default since we don't specify the "index_cls", a simple vector
store index is initialized.
   reader = WikipediaReader()
   tool = OnDemandLoaderTool.from_defaults(
       reader,
       name="Wikipedia Tool",
       description="A tool for loading and querying articles from Wikipedia",
   )
Testing
We can try running the tool by itself (or as a LangChain tool), just
to showcase what the interface is like!
Note that besides the arguments required for the data loader, the tool
also takes in a "query_str" which will be the query against the index.
   # run tool by itself
   tool(["Berlin"], query_str="What's the arts and culture scene in Berlin?")
   "\nBerlin has a vibrant and diverse arts and culture scene. It is home to 44 theaters and stages, three major opera houses, and numerous art galleries. The cityscape of Berlin displays large quantities of urban street art, and the Berlin Wall has become one of the largest open-air canvasses in the world. Berlin also has a long history of gay culture, and is an important birthplace of the LGBT rights movement. There are many festivals and events throughout the year, such as the Berlin International Film Festival, the Karneval der Kulturen, the Berlin Festival, and the New Year's Eve celebrations. The city is also home to many museums, such as the Museum Island, the Gemäldegalerie, the Neue Nationalgalerie, the Pergamon Museum, the Bode Museum, the Hamburger Bahnhof, the German Museum of Technology, the Jewish Museum, the Museum für Naturkunde, the Kupferstichkabinett Berlin, the Museum Berggruen, and the Beate Uhse Erotic Museum."
   # run tool as langchain structured tool
   lc_tool = tool.to_langchain_structured_tool(verbose=True)
   lc_tool.run(
       tool_input={
           "pages": ["Berlin"],
           "query_str": "What's the arts and culture scene in Berlin?",
       }
   )
Initialize LangChain Agent
For tutorial purposes, the agent just has access to one tool - the
Wikipedia Reader
Note that we need to use Structured Tools from LangChain.
   from langchain.agents import initialize_agent
   from langchain.chat_models import ChatOpenAI
   llm = ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo", streaming=True)
   agent = initialize_agent(
       [lc_tool],
       llm=llm,
       agent="structured-chat-zero-shot-react-description",
       verbose=True,
   )
Now let's run some queries!
The OnDemandLoaderTool allows the agent to simultaneously 1) load the
data from Wikipedia, 2) query that data.
   agent.run("Tell me about the arts and culture of Berlin")
   Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised APIConnectionError: Error communicating with OpenAI: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')).

OnDemandLoaderTool Tutorial

   > Entering new AgentExecutor chain...
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Culture in Berlin"],
       "query_str": "What is the arts and culture scene like in Berlin?"
     }
   }
   ```

   Observation:
   The arts and culture scene in Berlin is vibrant and diverse. The city is home to over 600 art galleries, 153 museums, and numerous cultural institutions. It is a world city of culture and creative industries, and is home to many international and regional television and radio stations. Berlin is also home to two major German-language publishing houses, and is an important center of the European and German film industry. The city is also known for its nightlife, with many clubs and festivals, such as the Berlin International Film Festival, the Karneval der Kulturen, and the Christopher Street Day. Berlin is also home to the largest gay fetish festivals in Europe.
   Thought:The human may want more specific information about certain aspects of Berlin's arts and culture scene.
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Culture in Berlin"],
       "query_str": "What are some notable museums in Berlin?"
     }
   }
   ```

   Observation:
   Some notable museums in Berlin include the Deutsches Historisches Museum, the Bauhaus Archive, the Jewish Museum, the German Museum of Technology, the Museum für Naturkunde, the Museum of Asian Art, the Ethnological Museum, the Museum of European Cultures, the Allied Museum, the Brücke Museum, the Stasi Museum, the Beate Uhse Erotic Museum, and the Pergamon Museum.
   Thought:The human may also be interested in learning about the music scene in Berlin.
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Music in Berlin"],
       "query_str": "What is the music scene like in Berlin?"
     }
   }
   ```

   Observation:
   The music scene in Berlin is vibrant and diverse. It is home to many nightclubs, including Kunst Haus Tacheles, Cookies, Tresor, WMF, Ufo, E-Werk, KitKatClub and Berghain, which are known for their long parties. It is also home to many concert music institutions, such as the Berlin Philharmonic Orchestra, the Konzerthausorchester Berlin, the Berlin Radio Symphony Orchestra, the Staatskapelle Berlin, and the SO36 in Kreuzberg. The city is also known for its influence on rock music, with bands like U2 recording at Hansa Studios near the Berlin Wall. Additionally, Berlin is home to many creative industries, such as music, film, advertising, architecture, art, design, fashion, performing arts, publishing, TV, radio, and video games. It is also home to many important musical figures, such as Johann Joachim Quantz, Carl Philipp Emanuel Bach, the Graun brothers, Wilhelm Friedemann Bach, Carl Friedrich Christian Fasch, Johann Friedrich Reichardt, Carl Friedrich Zelter, Friedrich Heinrich Himmel, Vincenzo Righini, Felix Mendelssohn Bartholdy, Spontini, Meyerbeer, Richard Strauss, Arnold Schoenberg, Friedrich Wilhelm Marpurg, Johann Philipp Kirnberger, Reichardt, E. T. A. Hoffmann, Ludwig Rellstab, and A. B. Marx. There are also three major opera houses in Berlin: the Deutsche Oper, the Berlin State Opera, and the Komische Oper.

OnDemandLoaderTool Tutorial

   Thought:The human may want to know more about the theater scene in Berlin.
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Theatre in Berlin"],
       "query_str": "What is the theater scene like in Berlin?"
     }
   }
   ```

   Observation:
   The theater scene in Berlin is vibrant and diverse. There are a variety of venues, from traditional theaters to modern cinemas, as well as a range of genres and styles. The Berlin Wintergarten theatre, which opened in 1887 and was destroyed during the Second World War, was the first Bioscop movie theater in history. The theatre was restarted, relocated and the title licensed in 1992, and is now located on Potsdamer Straße just South of Potsdamer Platz in Berlin. There are also many other theaters in the city, including the Berliner Ensemble, the Volksbühne, and the Schaubühne.
   Thought:The human may want to know more about the street art scene in Berlin.
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Street art in Berlin"],
       "query_str": "What is the street art scene like in Berlin?"
     }
   }
   ```

   Observation:
   The street art scene in Berlin is vibrant and diverse. It has been home to street artists such as Thierry Noir Tavar Zawacki a.k.a. ABOVE and SP 38, and post-communism, cheap rents, and ramshackle buildings have given rise to street art in areas such as Mitte, Prenzlauer Berg, Kreuzberg, and Friedrichshain. In 2016, StreetArtNews initiated an urban artwork in the name of Urban Nation Berlin, in which several famous artists participated. Street art by Bleepsgr, whose work has been categorized as "artivism", can be found in neighborhoods such as Psiri.
   Thought:The human may want to know more about the film industry in Berlin.
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Cinema of Germany"],
       "query_str": "What is the film industry like in Berlin?"
     }
   }
   ```

   Observation:
   The film industry in Berlin is thriving and has a long history. It is home to the Berlin International Film Festival, the Deutsche Filmakademie, and several film schools. Berlin is also home to many prominent personalities in the film industry, such as Dieter Kosslick, director of the Berlin International Film Festival, and Fritz Lang, a renowned director. The city is also home to several production companies, and is a major hub for the German film industry. Berlin is known for its diverse range of films, from silent films to contemporary works, and is a major center for the production of both feature films and television series.
   Thought:The human may want to know more about the literature scene in Berlin.
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Literature in Berlin"],
       "query_str": "What is the literature scene like in Berlin?"
     }
   }
   ```

OnDemandLoaderTool Tutorial

   Observation:
   The literature scene in Berlin is quite diverse and vibrant. There are a variety of literary genres represented in the city, from poetry to prose to children's literature. Berlin is home to a number of literary festivals, book fairs, and other events that celebrate the written word. There are also a number of independent bookstores, libraries, and other literary institutions that promote the reading and writing of literature. Berlin is also home to a number of renowned authors, including Nobel Prize winners Günter Grass and Herta Müller.
   Thought:The human may want to know more about the architecture scene in Berlin.
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Architecture in Berlin"],
       "query_str": "What is the architecture scene like in Berlin?"
     }
   }
   ```

   Observation:
   Berlin's architecture scene is incredibly diverse and eclectic. The city has been shaped by its history, with each of the governments based in Berlin initiating ambitious construction programs that have left their distinct mark on the city. There are many Plattenbauten in Eastern Berlin, as well as the iconic East Side Gallery, Fernsehturm, Gendarmenmarkt, Museum Island, Unter den Linden, Brandenburg Gate, Potsdamer Platz, Hackescher Markt, Straße des 17. Juni, Kurfürstendamm, Schloss Bellevue, and Funkturm Berlin. These landmarks are a mix of classical, modern, and postmodern architecture, and many of them have been restored after suffering damage during World War II.
   Thought:The human may want to know more about the fashion scene in Berlin.
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Fashion in Berlin"],
       "query_str": "What is the fashion scene like in Berlin?"
     }
   }
   ```

   Observation:
   The fashion scene in Berlin is vibrant and creative, with many young designers flourishing in the fashion capital. Mercedes-Benz is the main sponsor of the fashion week, which takes place twice a year in January and July. There are a variety of fashion fairs, such as BREAD & BUTTER, Premium Fair, Bright Tradeshow, (capsule), Show&Order, PanoramaBerlin and The Gallery Berlin. The StyleNite by Berlin-based designer Michael Michalsky is a popular event, featuring unusual performances of different art disciplines combined with state-of-the-art fashion. Models of all ages and abilities are featured in the shows, including disabled models and models aged over 60.
   Thought:The human may want to know more about the food scene in Berlin.
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Cuisine of Berlin"],
       "query_str": "What is the food scene like in Berlin?"
     }
   }
   ```

   Observation:
   The food scene in Berlin is very diverse and international. It is home to a wide variety of cuisines, including German, Turkish, Arab, Vietnamese, Chinese, Thai, Indian, Korean, Japanese, Spanish, Italian, and Greek. There are numerous restaurants, pubs, bakeries, and delicatessen markets, as well as fast-food versions of the doner kebab sandwich. Berlin is also well known for its vegetarian and vegan cuisine, innovative food scene, pop-up street food markets, supper clubs, and food festivals. Additionally, there are seven restaurants that have been awarded two Michelin stars and 14 restaurants that have been awarded one Michelin star.

OnDemandLoaderTool Tutorial

   Thought:The human may want to know more about the dance scene in Berlin.
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["Dance in Germany"],
       "query_str": "What is the dance scene like in Berlin?"
     }
   }
   ```

   /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages/wikipedia/wikipedia.py:389: GuessedAtParserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.
   The code that caused this warning is on line 389 of the file /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages/wikipedia/wikipedia.py. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.
     lis = BeautifulSoup(html).find_all('li')
   ---------------------------------------------------------------------------
   DisambiguationError                       Traceback (most recent call last)
   Cell In[12], line 1
   ----> 1 agent.run("Tell me about the arts and culture of Berlin")
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/langchain/chains/base.py:236, in Chain.run(self, callbacks, *args, **kwargs)
       234     if len(args) != 1:
       235         raise ValueError("`run` supports only one positional argument.")
   --> 236     return self(args[0], callbacks=callbacks)[self.output_keys[0]]
       238 if kwargs and not args:
       239     return self(kwargs, callbacks=callbacks)[self.output_keys[0]]
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/langchain/chains/base.py:140, in Chain.__call__(self, inputs, return_only_outputs, callbacks)
       138 except (KeyboardInterrupt, Exception) as e:
       139     run_manager.on_chain_error(e)
   --> 140     raise e
       141 run_manager.on_chain_end(outputs)
       142 return self.prep_outputs(inputs, outputs, return_only_outputs)
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/langchain/chains/base.py:134, in Chain.__call__(self, inputs, return_only_outputs, callbacks)
       128 run_manager = callback_manager.on_chain_start(
       129     {"name": self.__class__.__name__},
       130     inputs,
       131 )
       132 try:
       133     outputs = (
   --> 134         self._call(inputs, run_manager=run_manager)
       135         if new_arg_supported
       136         else self._call(inputs)
       137     )
       138 except (KeyboardInterrupt, Exception) as e:
       139     run_manager.on_chain_error(e)
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/langchain/agents/agent.py:951, in AgentExecutor._call(self, inputs, run_manager)
       949 # We now enter the agent loop (until it returns something).
       950 while self._should_continue(iterations, time_elapsed):
   --> 951     next_step_output = self._take_next_step(
       952         name_to_tool_map,
       953         color_mapping,
       954         inputs,
       955         intermediate_steps,
       956         run_manager=run_manager,
       957     )
       958     if isinstance(next_step_output, AgentFinish):

OnDemandLoaderTool Tutorial

       959         return self._return(
       960             next_step_output, intermediate_steps, run_manager=run_manager
       961         )
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/langchain/agents/agent.py:818, in AgentExecutor._take_next_step(self, name_to_tool_map, color_mapping, inputs, intermediate_steps, run_manager)
       816         tool_run_kwargs["llm_prefix"] = ""
       817     # We then call the tool on the tool input to get an observation
   --> 818     observation = tool.run(
       819         agent_action.tool_input,
       820         verbose=self.verbose,
       821         color=color,
       822         callbacks=run_manager.get_child() if run_manager else None,
       823         **tool_run_kwargs,
       824     )
       825 else:
       826     tool_run_kwargs = self.agent.tool_run_logging_kwargs()
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/langchain/tools/base.py:255, in BaseTool.run(self, tool_input, verbose, start_color, color, callbacks, **kwargs)
       253 except (Exception, KeyboardInterrupt) as e:
       254     run_manager.on_tool_error(e)
   --> 255     raise e
       256 run_manager.on_tool_end(str(observation), color=color, name=self.name, **kwargs)
       257 return observation
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/langchain/tools/base.py:249, in BaseTool.run(self, tool_input, verbose, start_color, color, callbacks, **kwargs)
       246 try:
       247     tool_args, tool_kwargs = self._to_args_and_kwargs(parsed_input)
       248     observation = (
   --> 249         self._run(*tool_args, run_manager=run_manager, **tool_kwargs)
       250         if new_arg_supported
       251         else self._run(*tool_args, **tool_kwargs)
       252     )
       253 except (Exception, KeyboardInterrupt) as e:
       254     run_manager.on_tool_error(e)
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/langchain/tools/base.py:436, in StructuredTool._run(self, run_manager, *args, **kwargs)
       427 """Use the tool."""
       428 new_argument_supported = signature(self.func).parameters.get("callbacks")
       429 return (
       430     self.func(
       431         *args,
       432         callbacks=run_manager.get_child() if run_manager else None,
       433         **kwargs,
       434     )
       435     if new_argument_supported
   --> 436     else self.func(*args, **kwargs)
       437 )
   File ~/Programming/gpt_index/llama_index/tools/ondemand_loader_tool.py:114, in OnDemandLoaderTool.__call__(self, *args, **kwargs)
       112 else:
       113     query_str = kwargs.pop(self._query_str_kwargs_key)
   --> 114 docs = self._reader.load_data(*args, **kwargs)
       115 index = self._index_cls.from_documents(docs, **self._index_kwargs)
       116 # TODO: add query kwargs
   File ~/Programming/gpt_index/llama_index/readers/wikipedia.py:35, in WikipediaReader.load_data(self, pages, **load_kwargs)
        33 results = []
        34 for page in pages:
   ---> 35     page_content = wikipedia.page(page, **load_kwargs).content
        36     results.append(Document(page_content))
        37 return results
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/wikipedia/wikipedia.py:276, in page(title, pageid, auto_suggest, redirect, preload)

OnDemandLoaderTool Tutorial

       273     except IndexError:
       274       # if there is no suggestion or search results, the page doesn't exist
       275       raise PageError(title)
   --> 276   return WikipediaPage(title, redirect=redirect, preload=preload)
       277 elif pageid is not None:
       278   return WikipediaPage(pageid=pageid, preload=preload)
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/wikipedia/wikipedia.py:299, in WikipediaPage.__init__(self, title, pageid, redirect, preload, original_title)
       296 else:
       297   raise ValueError("Either a title or a pageid must be specified")
   --> 299 self.__load(redirect=redirect, preload=preload)
       301 if preload:
       302   for prop in ('content', 'summary', 'images', 'references', 'links', 'sections'):
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/wikipedia/wikipedia.py:393, in WikipediaPage.__load(self, redirect, preload)
       390   filtered_lis = [li for li in lis if not 'tocsection' in ''.join(li.get('class', []))]
       391   may_refer_to = [li.a.get_text() for li in filtered_lis if li.a]
   --> 393   raise DisambiguationError(getattr(self, 'title', page['title']), may_refer_to)
       395 else:
       396   self.pageid = pageid
   DisambiguationError: "Dance, Dance, Dance" may refer to:
   "Dance, Dance, Dance" (The Beach Boys song)
   "Dance, Dance, Dance" (Neil Young song)
   "Dance, Dance, Dance" (Yowsah, Yowsah, Yowsah)
   "Dance Dance Dance" (James Cottriall song)
   "Dance Dance Dance" (E-girls song)
   Dance Dance Dance/My Lady
   soundtrack
   Why Do You Have to Go/Dance, Dance, Dance
   Youth Novels
   Fly Like an Eagle
   Dance Dance Dance (German TV series)
   Dance Dance Dance (British TV series)
   Dance Dance Dance (novel)
   Dance, Dance, Dance: The Best of Chic
   Dance, Dance (disambiguation)
   agent.run("Tell me about the critical reception to The Departed")
   > Entering new AgentExecutor chain...
   Action:
   ```
   {
     "action": "Wikipedia Tool",
     "action_input": {
       "pages": ["The Departed"],
       "query_str": "critical reception"
     }
   }
   ```

   Observation:
   The critical reception of The Departed was overwhelmingly positive. On review aggregator Rotten Tomatoes, the film holds a 91% approval rating based on 284 reviews, with an average rating of 8.3/10. The website's critics consensus reads, "Featuring outstanding work from an excellent cast, The Departed is a thoroughly engrossing gangster drama with the gritty authenticity and soupy morality we have come to expect from Martin Scorsese." Metacritic, which uses a weighted average, assigned the film a score of 85 out of 100 based on 39 critics, indicating "universal acclaim". Audiences polled by CinemaScore gave the film an average grade of "A−" on an A+ to F scale. Entertainment Weekly ranked it on its end-of-the-decade "Best of" list, saying: "If they're lucky, directors make one classic film in their career. Martin Scorsese has one per decade (Taxi Driver in the '70s, Raging Bull in the '80s, Goodfellas in the '90s). His 2006 Irish Mafia masterpiece kept the streak alive." Roger Ebert gave the film four stars out of four, praising Scorsese for thematically differentiating his film from the original. Online critic James Berardinelli awarded the film four stars out of four, praising it as "an American epic tragedy." He went on to claim that the film deserves to be ranked alongside Scorsese's past successes, including Taxi Driver, Raging Bull and Goodfellas.

OnDemandLoaderTool Tutorial

   Thought:The critical reception to The Departed was very positive.
   Action:
   ```
   {
     "action": "Final Answer",
     "action_input": "The critical reception to The Departed was overwhelmingly positive, with an approval rating of 91% on Rotten Tomatoes and a score of 85 out of 100 on Metacritic. It was praised for its outstanding cast, gritty authenticity, and soupy morality. Many critics ranked it alongside Scorsese's past successes, including Taxi Driver, Raging Bull, and Goodfellas."
   }
   ```

   > Finished chain.
   "The critical reception to The Departed was overwhelmingly positive, with an approval rating of 91% on Rotten Tomatoes and a score of 85 out of 100 on Metacritic. It was praised for its outstanding cast, gritty authenticity, and soupy morality. Many critics ranked it alongside Scorsese's past successes, including Taxi Driver, Raging Bull, and Goodfellas."


RunGPT

RunGPT is an open-source cloud-native large-scale multimodal models
(LMMs) serving framework. It is designed to simplify the deployment
and management of large language models, on a distributed cluster of
GPUs. RunGPT aim to make it a one-stop solution for a centralized and
accessible place to gather techniques for optimizing large-scale
multimodal models and make them easy to use for everyone. In RunGPT,
we have supported a number of LLMs such as LLaMA, Pythia, StableLM,
Vicuna, MOSS, and Large Multi-modal Model(LMMs) like MiniGPT-4 and
OpenFlamingo additionally.
Setup
Firstly, you need to install rungpt package in your python environment
with "pip install"
   !pip install rungpt
After installing successfully, models supported by RunGPT can be
deployed with an one-line command. This option will download target
language model from open source platform and deploy it as a service at
a localhost port, which can be accessed by http or grpc requests. I
suppose you not run this command in jupyter book, but in command line
instead.
   !rungpt serve decapoda-research/llama-7b-hf --precision fp16 --device_map balanced
Basic Usage
Call "complete" with a prompt
   from llama_index.llms.rungpt import RunGptLLM
   llm = RunGptLLM()
   promot = "What public transportation might be available in a city?"
   response = llm.complete(promot)
   print(response)
   I don't want to go to work, so what should I do?
   I have a job interview on Monday. What can I wear that will make me look professional but not too stuffy or boring?
Call "chat" with a list of messages
   from llama_index.llms.base import ChatMessage, MessageRole
   from llama_index.llms.rungpt import RunGptLLM
   messages = [
       ChatMessage(
           role=MessageRole.USER,
           content="Now, I want you to do some math for me.",
       ),
       ChatMessage(role=MessageRole.ASSISTANT, content="Sure, I would like to help you."),
       ChatMessage(
           role=MessageRole.USER, content="How many points determine a straight line?"
       ),
   ]
   llm = RunGptLLM()
   response = llm.chat(messages=messages, temperature=0.8, max_tokens=15)
   print(response)
Streaming
Using "stream_complete" endpoint
   promot = "What public transportation might be available in a city?"
   response = RunGptLLM().stream_complete(promot)
   for item in response:
       print(item.text)
Using "stream_chat" endpoint
   from llama_index.llms.rungpt import RunGptLLM
   messages = [
       ChatMessage(
           role=MessageRole.USER,
           content="Now, I want you to do some math for me.",
       ),
       ChatMessage(role=MessageRole.ASSISTANT, content="Sure, I would like to help you."),
       ChatMessage(
           role=MessageRole.USER, content="How many points determine a straight line?"
       ),
   ]
   response = RunGptLLM().stream_chat(messages=messages)
   for item in response:
       print(item.message)