Multi-Document Agents (V1)

In this guide, you learn towards setting up a multi-document agent
over the LlamaIndex documentation.
This is an extension of V0 multi-document agents with the additional
features:
* Reranking during document (tool) retrieval
* Query planning tool that the agent can use to plan
We do this with the following architecture:
* setup a "document agent" over each Document: each doc agent can do
  QA/summarization within its doc
* setup a top-level agent over this set of document agents. Do tool
  retrieval and then do CoT over the set of tools to answer a
  question.
   %load_ext autoreload
   %autoreload 2
Setup and Download Data
In this section, we'll load in the LlamaIndex documentation.
   domain = "docs.llamaindex.ai"
   docs_url = "https://docs.llamaindex.ai/en/latest/"
   !wget -e robots=off --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains {domain} --no-parent {docs_url}
   from llama_hub.file.unstructured.base import UnstructuredReader
   from pathlib import Path
   from llama_index.llms import OpenAI
   from llama_index import ServiceContext
   reader = UnstructuredReader()
   [nltk_data] Downloading package punkt to /Users/jerryliu/nltk_data...
   [nltk_data]   Package punkt is already up-to-date!
   [nltk_data] Downloading package averaged_perceptron_tagger to
   [nltk_data]     /Users/jerryliu/nltk_data...
   [nltk_data]   Package averaged_perceptron_tagger is already up-to-
   [nltk_data]       date!
   all_files_gen = Path("./docs.llamaindex.ai/").rglob("*")
   all_files = [f.resolve() for f in all_files_gen]
   all_html_files = [f for f in all_files if f.suffix.lower() == ".html"]
   len(all_html_files)
   418
   from llama_index import Document
   # TODO: set to higher value if you want more docs
   doc_limit = 100
   docs = []
   for idx, f in enumerate(all_html_files):
       if idx > doc_limit:
           break
       print(f"Idx {idx}/{len(all_html_files)}")
       loaded_docs = reader.load_data(file=f, split_documents=True)
       # Hardcoded Index. Everything before this is ToC for all pages
       start_idx = 72
       loaded_doc = Document(
           text="\n\n".join([d.get_content() for d in loaded_docs[72:]]),
           metadata={"path": str(f)},
       )
       print(loaded_doc.metadata["path"])
       docs.append(loaded_doc)
Define LLM + Service Context + Callback Manager
   llm = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(llm=llm)
Building Multi-Document Agents
In this section we show you how to construct the multi-document agent.
We first build a document agent for each document, and then define the
top-level parent agent with an object index.
   from llama_index import VectorStoreIndex, SummaryIndex
   import nest_asyncio
   nest_asyncio.apply()
Build Document Agent for each Document
In this section we define "document agents" for each document.
We define both a vector index (for semantic search) and summary index
(for summarization) for each document. The two query engines are then
converted into tools that are passed to an OpenAI function calling
agent.
This document agent can dynamically choose to perform semantic search
or summarization within a given document.

Multi-Document Agents (V1)

We create a separate document agent for each city.
   from llama_index.agent import OpenAIAgent
   from llama_index import load_index_from_storage, StorageContext
   from llama_index.tools import QueryEngineTool, ToolMetadata
   from llama_index.node_parser import SimpleNodeParser
   import os
   from tqdm.notebook import tqdm
   import pickle
   async def build_agent_per_doc(nodes, file_base):
       print(file_base)
       vi_out_path = f"./data/llamaindex_docs/{file_base}"
       summary_out_path = f"./data/llamaindex_docs/{file_base}_summary.pkl"
       if not os.path.exists(vi_out_path):
           Path("./data/llamaindex_docs/").mkdir(parents=True, exist_ok=True)
           # build vector index
           vector_index = VectorStoreIndex(nodes, service_context=service_context)
           vector_index.storage_context.persist(persist_dir=vi_out_path)
       else:
           vector_index = load_index_from_storage(
               StorageContext.from_defaults(persist_dir=vi_out_path),
               service_context=service_context,
           )
       # build summary index
       summary_index = SummaryIndex(nodes, service_context=service_context)
       # define query engines
       vector_query_engine = vector_index.as_query_engine()
       summary_query_engine = summary_index.as_query_engine(response_mode="tree_summarize")
       # extract a summary
       if not os.path.exists(summary_out_path):
           Path(summary_out_path).parent.mkdir(parents=True, exist_ok=True)
           summary = str(
               await summary_query_engine.aquery(
                   "Extract a concise 1-2 line summary of this document"
               )
           )
           pickle.dump(summary, open(summary_out_path, "wb"))
       else:
           summary = pickle.load(open(summary_out_path, "rb"))
       # define tools
       query_engine_tools = [
           QueryEngineTool(
               query_engine=vector_query_engine,
               metadata=ToolMetadata(
                   name=f"vector_tool_{file_base}",
                   description=f"Useful for questions related to specific facts",
               ),
           ),
           QueryEngineTool(
               query_engine=summary_query_engine,
               metadata=ToolMetadata(
                   name=f"summary_tool_{file_base}",
                   description=f"Useful for summarization questions",
               ),
           ),
       ]
       # build agent
       function_llm = OpenAI(model="gpt-4")
       agent = OpenAIAgent.from_tools(
           query_engine_tools,
           llm=function_llm,
           verbose=True,
           system_prompt=f"""\
   You are a specialized agent designed to answer queries about the `{file_base}.html` part of the LlamaIndex docs.
   You must ALWAYS use at least one of the tools provided when answering a question; do NOT rely on prior knowledge.\
   """,
       )
       return agent, summary
   async def build_agents(docs):
       node_parser = SimpleNodeParser.from_defaults()
       # Build agents dictionary
       agents_dict = {}
       extra_info_dict = {}
       # # this is for the baseline
       # all_nodes = []
       for idx, doc in enumerate(tqdm(docs)):
           nodes = node_parser.get_nodes_from_documents([doc])
           # all_nodes.extend(nodes)
           # ID will be base + parent
           file_path = Path(doc.metadata["path"])
           file_base = str(file_path.parent.stem) + "_" + str(file_path.stem)
           agent, summary = await build_agent_per_doc(nodes, file_base)
           agents_dict[file_base] = agent
           extra_info_dict[file_base] = {"summary": summary, "nodes": nodes}
       return agents_dict, extra_info_dict
   agents_dict, extra_info_dict = await build_agents(docs)
Build Retriever-Enabled OpenAI Agent
We build a top-level agent that can orchestrate across the different
document agents to answer any user query.

Multi-Document Agents (V1)

This "RetrieverOpenAIAgent" performs tool retrieval before tool use
(unlike a default agent that tries to put all tools in the prompt).
**Improvements from V0**: We make the following improvements compared
to the "base" version in V0.
* Adding in reranking: we use Cohere reranker to better filter the
  candidate set of documents.
* Adding in a query planning tool: we add an explicit query planning
  tool that's dynamically created based on the set of retrieved tools.
   # define tool for each document agent
   all_tools = []
   for file_base, agent in agents_dict.items():
       summary = extra_info_dict[file_base]["summary"]
       doc_tool = QueryEngineTool(
           query_engine=agent,
           metadata=ToolMetadata(
               name=f"tool_{file_base}",
               description=summary,
           ),
       )
       all_tools.append(doc_tool)
   print(all_tools[0].metadata)
   ToolMetadata(description='LlamaIndex is a data framework that allows LLM applications to ingest, structure, and access private or domain-specific data by providing tools such as data connectors, data indexes, engines, data agents, and application integrations. It is designed for beginners, advanced users, and everyone in between, and offers both high-level and lower-level APIs for customization. LlamaIndex can be installed using pip and has detailed documentation and tutorials available. It is available on GitHub and PyPi, and there is also a Typescript package available. The LlamaIndex community can be joined on Twitter and Discord.', name='tool_latest_index', fn_schema=<class 'llama_index.tools.types.DefaultToolFnSchema'>)
   # define an "object" index and retriever over these tools
   from llama_index import VectorStoreIndex
   from llama_index.objects import ObjectIndex, SimpleToolNodeMapping, ObjectRetriever
   from llama_index.retrievers import BaseRetriever
   from llama_index.indices.postprocessor import CohereRerank
   from llama_index.tools import QueryPlanTool
   from llama_index.query_engine import SubQuestionQueryEngine
   from llama_index.llms import OpenAI
   llm = OpenAI(model_name="gpt-4-0613")
   tool_mapping = SimpleToolNodeMapping.from_objects(all_tools)
   obj_index = ObjectIndex.from_objects(
       all_tools,
       tool_mapping,
       VectorStoreIndex,
   )
   vector_node_retriever = obj_index.as_node_retriever(similarity_top_k=10)
   # define a custom retriever with reranking
   class CustomRetriever(BaseRetriever):
       def __init__(self, vector_retriever, postprocessor=None):
           self._vector_retriever = vector_retriever
           self._postprocessor = postprocessor or CohereRerank(top_n=5)
       def _retrieve(self, query_bundle):
           retrieved_nodes = self._vector_retriever.retrieve(query_bundle)
           filtered_nodes = self._postprocessor.postprocess_nodes(
               retrieved_nodes, query_bundle=query_bundle
           )
           return filtered_nodes
   # define a custom object retriever that adds in a query planning tool
   class CustomObjectRetriever(ObjectRetriever):
       def __init__(self, retriever, object_node_mapping, all_tools, llm=None):
           self._retriever = retriever
           self._object_node_mapping = object_node_mapping
           self._llm = llm or OpenAI("gpt-4-0613")
       def retrieve(self, query_bundle):
           nodes = self._retriever.retrieve(query_bundle)
           tools = [self._object_node_mapping.from_node(n.node) for n in nodes]
           sub_question_sc = ServiceContext.from_defaults(llm=self._llm)
           sub_question_engine = SubQuestionQueryEngine.from_defaults(
               query_engine_tools=tools, service_context=sub_question_sc

Multi-Document Agents (V1)

           )
           sub_question_description = f"""\
   Useful for any queries that involve comparing multiple documents. ALWAYS use this tool for comparison queries - make sure to call this \
   tool with the original query. Do NOT use the other tools for any queries involving multiple documents.
   """
           sub_question_tool = QueryEngineTool(
               query_engine=sub_question_engine,
               metadata=ToolMetadata(
                   name="compare_tool", description=sub_question_description
               ),
           )
           return tools + [sub_question_tool]
   custom_node_retriever = CustomRetriever(vector_node_retriever)
   # wrap it with ObjectRetriever to return objects
   custom_obj_retriever = CustomObjectRetriever(
       custom_node_retriever, tool_mapping, all_tools, llm=llm
   )
   tmps = custom_obj_retriever.retrieve("hello")
   print(len(tmps))
   6
   from llama_index.agent import FnRetrieverOpenAIAgent, ReActAgent
   top_agent = FnRetrieverOpenAIAgent.from_retriever(
       custom_obj_retriever,
       system_prompt=""" \
   You are an agent designed to answer queries about the documentation.
   Please always use the tools provided to answer a question. Do not rely on prior knowledge.\
   """,
       llm=llm,
       verbose=True,
   )
   # top_agent = ReActAgent.from_tools(
   #     tool_retriever=custom_obj_retriever,
   #     system_prompt=""" \
   # You are an agent designed to answer queries about the documentation.
   # Please always use the tools provided to answer a question. Do not rely on prior knowledge.\
   # """,
   #     llm=llm,
   #     verbose=True,
   # )
Define Baseline Vector Store Index
As a point of comparison, we define a "naive" RAG pipeline which dumps
all docs into a single vector index collection.
We set the top_k = 4
   all_nodes = [n for extra_info in extra_info_dict.values() for n in extra_info["nodes"]]
   base_index = VectorStoreIndex(all_nodes)
   base_query_engine = base_index.as_query_engine(similarity_top_k=4)
Running Example Queries
Let's run some example queries, ranging from QA / summaries over a
single document to QA / summarization over multiple documents.
   response = top_agent.query(
       "Tell me about the different types of evaluation in LlamaIndex"
   )
   === Calling Function ===
   Calling function: tool_api_reference_evaluation with args: {
     "input": "types of evaluation"
   }
   === Calling Function ===
   Calling function: vector_tool_api_reference_evaluation with args: {
     "input": "types of evaluation"
   }
   Got output: The types of evaluation can include correctness evaluation, faithfulness evaluation, guideline evaluation, hit rate evaluation, MRR (Mean Reciprocal Rank) evaluation, pairwise comparison evaluation, relevancy evaluation, and response evaluation.
   ========================
   Got output: The types of evaluation mentioned in the `api_reference_evaluation.html` part of the LlamaIndex docs include:
   1. Correctness Evaluation
   2. Faithfulness Evaluation
   3. Guideline Evaluation
   4. Hit Rate Evaluation
   5. MRR (Mean Reciprocal Rank) Evaluation
   6. Pairwise Comparison Evaluation
   7. Relevancy Evaluation
   8. Response Evaluation
   ========================
   print(response)
   There are several types of evaluation in LlamaIndex:
   1. Correctness Evaluation: This type of evaluation measures the accuracy of the retrieval results. It checks if the retrieved documents are correct and relevant to the query.
   2. Faithfulness Evaluation: Faithfulness evaluation measures how faithfully the retrieved documents represent the original data. It checks if the retrieved documents accurately reflect the information in the original documents.

Multi-Document Agents (V1)

   3. Guideline Evaluation: Guideline evaluation involves comparing the retrieval results against a set of guidelines or ground truth. It checks if the retrieval results align with the expected or desired outcomes.
   4. Hit Rate Evaluation: Hit rate evaluation measures the percentage of queries that return at least one relevant document. It is a binary evaluation metric that indicates the effectiveness of the retrieval system in finding relevant documents.
   5. MRR (Mean Reciprocal Rank) Evaluation: MRR evaluation measures the average rank of the first relevant document in the retrieval results. It provides a single value that represents the effectiveness of the retrieval system in ranking relevant documents.
   6. Pairwise Comparison Evaluation: Pairwise comparison evaluation involves comparing the retrieval results of different systems or algorithms. It helps determine which system performs better in terms of retrieval accuracy and relevance.
   7. Relevancy Evaluation: Relevancy evaluation measures the relevance of the retrieved documents to the query. It can be done using various metrics such as precision, recall, and F1 score.
   8. Response Evaluation: Response evaluation measures the quality of the response generated by the retrieval system. It checks if the response is informative, accurate, and helpful to the user.
   These evaluation types help assess the performance and effectiveness of the retrieval system in LlamaIndex.
   # baseline
   response = base_query_engine.query(
       "Tell me about the different types of evaluation in LlamaIndex"
   )
   print(str(response))
   LlamaIndex utilizes various types of evaluation methods to assess its performance and effectiveness. These evaluation methods include RelevancyEvaluator, RetrieverEvaluator, SemanticSimilarityEvaluator, PairwiseComparisonEvaluator, CorrectnessEvaluator, FaithfulnessEvaluator, and GuidelineEvaluator. Each of these evaluators serves a specific purpose in evaluating different aspects of the LlamaIndex system.
   response = top_agent.query(
       "Compare the content in the contributions page vs. index page."
   )
   === Calling Function ===
   Calling function: compare_tool with args: {
     "input": "content in the contributions page vs. index page"
   }
   Generated 2 sub questions.
   [1;3;38;2;237;90;200m[tool_development_contributing] Q: What is the content of the contributions page?
   [1;3;38;2;90;149;237m[tool_latest_index] Q: What is the content of the index page?
   === Calling Function ===
   Calling function: summary_tool_development_contributing with args: {
     "input": "development_contributing.html"
   }
   === Calling Function ===
   Calling function: vector_tool_latest_index with args: {
     "input": "content of the index page"
   }
   Got output: The development_contributing.html file provides information on how to contribute to LlamaIndex. It includes guidelines on what to work on, such as extending core modules, fixing bugs, adding usage examples, adding experimental features, and improving code quality and documentation. The file also provides details on each module, including data loaders, node parsers, text splitters, document/index/KV stores, managed index, vector stores, retrievers, query engines, query transforms, token usage optimizers, node postprocessors, and output parsers. Additionally, the file includes a development guideline section that covers environment setup, validating changes, formatting/linting, testing, creating example notebooks, and creating a pull request.
   ========================
   Got output: The content of the index page provides information about LlamaIndex, a data framework for LLM applications. It explains why LlamaIndex is useful for augmenting LLM models with private or domain-specific data that may be distributed across different applications and data stores. LlamaIndex offers tools such as data connectors, data indexes, engines, and data agents to ingest, structure, and access data. It is designed for beginners as well as advanced users who can customize and extend its modules. The page also provides installation instructions, tutorials, and links to the LlamaIndex ecosystem and associated projects.

Multi-Document Agents (V1)

   ========================
   [1;3;38;2;90;149;237m[tool_latest_index] A: The content of the `latest_index.html` page provides comprehensive information about LlamaIndex, a data framework for LLM applications. It explains the utility of LlamaIndex in augmenting LLM models with private or domain-specific data that may be distributed across different applications and data stores.
   The page details the tools offered by LlamaIndex, such as data connectors, data indexes, engines, and data agents, which are used to ingest, structure, and access data. It is designed to cater to both beginners and advanced users, with the flexibility to customize and extend its modules.
   Additionally, the page provides installation instructions and tutorials for users. It also includes links to the LlamaIndex ecosystem and associated projects for further exploration and understanding.
   [1;3;38;2;237;90;200m[tool_development_contributing] A: The `development_contributing.html` page of the LlamaIndex docs provides comprehensive information on how to contribute to the project. It includes guidelines on the areas to focus on, such as extending core modules, fixing bugs, adding usage examples, adding experimental features, and improving code quality and documentation.
   The page also provides detailed information on each module, including data loaders, node parsers, text splitters, document/index/KV stores, managed index, vector stores, retrievers, query engines, query transforms, token usage optimizers, node postprocessors, and output parsers.
   In addition, there is a development guideline section that covers various aspects of the development process, including environment setup, validating changes, formatting/linting, testing, creating example notebooks, and creating a pull request.
   Got output: The content in the contributions page of the LlamaIndex documentation provides comprehensive information on how to contribute to the project, including guidelines on areas to focus on and detailed information on each module. It also covers various aspects of the development process.
   On the other hand, the content in the index page of the LlamaIndex documentation provides comprehensive information about LlamaIndex itself, explaining its utility in augmenting LLM models with private or domain-specific data. It details the tools offered by LlamaIndex and provides installation instructions, tutorials, and links to the LlamaIndex ecosystem and associated projects.
   ========================
   print(response)
   The contributions page of the LlamaIndex documentation provides guidelines for contributing to LlamaIndex, including extending core modules, fixing bugs, adding usage examples, adding experimental features, and improving code quality and documentation. It also includes information on the environment setup, validating changes, formatting and linting, testing, creating example notebooks, and creating a pull request.
   On the other hand, the index page of the LlamaIndex documentation provides information about LlamaIndex itself. It explains that LlamaIndex is a data framework that allows LLM applications to ingest, structure, and access private or domain-specific data. It provides tools such as data connectors, data indexes, engines, data agents, and application integrations. The index page also mentions that LlamaIndex is designed for beginners, advanced users, and everyone in between, and offers both high-level and lower-level APIs for customization. It provides installation instructions, links to the GitHub and PyPi repositories, and information about the LlamaIndex community on Twitter and Discord.
   In summary, the contributions page focuses on contributing to LlamaIndex, while the index page provides an overview of LlamaIndex and its features.
   response = top_agent.query(
       "Can you compare the tree index and list index at a very high-level?"
   )
   print(str(response))
   At a high level, the Tree Index and List Index are two different types of indexes used in the system.
   The Tree Index is a tree-structured index that is built specifically for each query. It allows for the construction of a query-specific tree from leaf nodes to return a response. The Tree Index is designed to provide a more optimized and efficient way of retrieving nodes based on a query.

Multi-Document Agents (V1)

   On the other hand, the List Index is a keyword table index that supports operations such as inserting and deleting documents, retrieving nodes based on a query, and refreshing the index with updated documents. The List Index is a simpler index that uses a keyword table approach for retrieval.
   Both indexes have their own advantages and use cases. The choice between them depends on the specific requirements and constraints of the system.


Context-Augmented OpenAI Agent

In this tutorial, we show you how to use our
"ContextRetrieverOpenAIAgent" implementation to build an agent on top
of OpenAI's function API and store/index an arbitrary number of tools.
Our indexing/retrieval modules help to remove the complexity of having
too many functions to fit in the prompt.
Initial Setup
Here we setup a ContextRetrieverOpenAIAgent. This agent will perform
retrieval first before calling any tools. This can help ground the
agent's tool picking and answering capabilities in context.
   import json
   from typing import Sequence
   from llama_index import (
       SimpleDirectoryReader,
       VectorStoreIndex,
       StorageContext,
       load_index_from_storage,
   )
   from llama_index.tools import QueryEngineTool, ToolMetadata
   try:
       storage_context = StorageContext.from_defaults(persist_dir="./storage/march")
       march_index = load_index_from_storage(storage_context)
       storage_context = StorageContext.from_defaults(persist_dir="./storage/june")
       june_index = load_index_from_storage(storage_context)
       storage_context = StorageContext.from_defaults(persist_dir="./storage/sept")
       sept_index = load_index_from_storage(storage_context)
       index_loaded = True
   except:
       index_loaded = False
   # build indexes across the three data sources
   if not index_loaded:
       # load data
       march_docs = SimpleDirectoryReader(
           input_files=["../data/10q/uber_10q_march_2022.pdf"]
       ).load_data()
       june_docs = SimpleDirectoryReader(
           input_files=["../data/10q/uber_10q_june_2022.pdf"]
       ).load_data()
       sept_docs = SimpleDirectoryReader(
           input_files=["../data/10q/uber_10q_sept_2022.pdf"]
       ).load_data()
       # build index
       march_index = VectorStoreIndex.from_documents(march_docs)
       june_index = VectorStoreIndex.from_documents(june_docs)
       sept_index = VectorStoreIndex.from_documents(sept_docs)
       # persist index
       march_index.storage_context.persist(persist_dir="./storage/march")
       june_index.storage_context.persist(persist_dir="./storage/june")
       sept_index.storage_context.persist(persist_dir="./storage/sept")
   march_engine = march_index.as_query_engine(similarity_top_k=3)
   june_engine = june_index.as_query_engine(similarity_top_k=3)
   sept_engine = sept_index.as_query_engine(similarity_top_k=3)
   query_engine_tools = [
       QueryEngineTool(
           query_engine=march_engine,
           metadata=ToolMetadata(
               name="uber_march_10q",
               description="Provides information about Uber 10Q filings for March 2022. "
               "Use a detailed plain text question as input to the tool.",
           ),
       ),
       QueryEngineTool(
           query_engine=june_engine,
           metadata=ToolMetadata(
               name="uber_june_10q",
               description="Provides information about Uber financials for June 2021. "
               "Use a detailed plain text question as input to the tool.",
           ),
       ),
       QueryEngineTool(
           query_engine=sept_engine,
           metadata=ToolMetadata(
               name="uber_sept_10q",
               description="Provides information about Uber financials for Sept 2021. "
               "Use a detailed plain text question as input to the tool.",
           ),
       ),
   ]
Try Context-Augmented Agent
Here we augment our agent with context in different settings:
* toy context: we define some abbreviations that map to financial
  terms (e.g. R=Revenue). We supply this as context to the agent
   from llama_index.schema import Document
   from llama_index.agent import ContextRetrieverOpenAIAgent

Context-Augmented OpenAI Agent

   # toy index - stores a list of abbreviations
   texts = [
       "Abbreviation: X = Revenue",
       "Abbreviation: YZ = Risk Factors",
       "Abbreviation: Z = Costs",
   ]
   docs = [Document(text=t) for t in texts]
   context_index = VectorStoreIndex.from_documents(docs)
   context_agent = ContextRetrieverOpenAIAgent.from_tools_and_retriever(
       query_engine_tools, context_index.as_retriever(similarity_top_k=1), verbose=True
   )
   response = context_agent.chat("What is the YZ of March 2022?")
   Context information is below.
   ---------------------
   Abbreviation: YZ = Risk Factors
   ---------------------
   Given the context information and not prior knowledge, either pick the corresponding tool or answer the function: What is the YZ of March 2022?
   === Calling Function ===
   Calling function: uber_march_10q with args: {
     "input": "Risk Factors"
   }
   Got output:
   •The COVID-19 pandemic and the impact of actions to mitigate the pandemic have adversely affected and may continue to adversely affect parts of our business.
   •Our business would be adversely affected if Drivers were classified as employees, workers or quasi-employees instead of independent contractors.
   •The mobility, delivery, and logistics industries are highly competitive, with well-established and low-cost alternatives that have been available for decades, low barriers to entry, low switching costs, and well-capitalized competitors in nearly every major geographic region.
   •To remain competitive in certain markets, we have in the past lowered, and may continue to lower, fares or service fees, and we have in the past offered, and may continue to offer, significant Driver incentives and consumer discounts and promotions.
   •We have incurred significant losses since inception, including in the United States and other major markets. We expect our operating expenses to increase significantly in the foreseeable future, and we may not achieve or maintain profitability.
   •If we are unable to attract or maintain a critical mass of Drivers, consumers, merchants, shippers, and carriers, whether as a result of competition or other factors, our platform will become less appealing to platform users.
   •Maintaining and enhancing our brand and reputation is critical to our business prospects. We have previously received significant media coverage and negative publicity regarding our brand and reputation, and while we have taken significant steps to rehabilitate our brand and reputation, failure to maintain and enhance our brand and reputation could adversely affect our business.
   •The impact of economic conditions, including the resulting effect on discretionary consumer spending, may harm our business and operating results.
   •Increases in fuel, food, labor, energy, and other costs due to inflation and other factors could adversely affect our operating results.
   •If we experience security or privacy breaches or other unauthorized or improper access to, use of, disclosure of, alteration of or destruction of our proprietary or confidential data, employee data, or platform user data.
   •Cyberattacks, including computer malware, ransomware, viruses, spamming, and phishing attacks could harm our reputation, business, and operating results.
   •We are subject to climate change risks, including physical and transitional risks, and if we are unable to manage such risks, our business may be adversely impacted.
   •We have made climate related commitments that require us to invest significant effort, resources, and management time and circumstances may arise, including those beyond our control, that may require us to revise the contemplated timeframes for implementing these commitments.
   •We rely on third parties maintaining open marketplaces to distribute our platform and to provide the software we use in certain of our products and offerings. If such third parties interfere with the distribution of our products or offerings or with our use of such software, our business would be adversely affected.

Context-Augmented OpenAI Agent

   •We will require additional capital to support the growth of our business, and this capital might not be available on reasonable terms or at all.
   •If we are unable to successfully identify, acquire and integrate suitable businesses, our operating results and prospects could be harmed, and any businesses we acquire may not perform as expected or be effectively integrated.
   •We may continue to be blocked from or limited in providing or operating our products and offerings in certain jurisdictions, and may be required to modify our business model in those jurisdictions as a result.
   •Our business is subject to numerous legal and regulatory risks that could have an adverse impact on our business and future prospects.
   •Our business is subject to extensive government regulation and oversight relating to the provision of payment and financial services.
   •We face risks related to our collection, use, transfer, disclosure, and other processing of data, which could result in investigations, inquiries, litigation, fines, legislative and regulatory action, and negative press about our privacy and data protection practices.
   •If we are unable to protect our intellectual property, or if third parties are successful in claiming that we are misappropriating the intellectual property of others, we may incur significant expense and our business may be adversely affected.
   •The market price of our common stock has been, and may continue to be, volatile or may decline steeply or suddenly regardless of our operating performance, and we may not be able to meet investor or analyst expectations. You may not be able to resell your shares at or above the price you paid and may lose all or part of your investment.
   ========================
   print(str(response))
   The risk factors for Uber in March 2022 include:
   1. The adverse impact of the COVID-19 pandemic and actions taken to mitigate it on Uber's business.
   2. The potential adverse effect on Uber's business if drivers are classified as employees instead of independent contractors.
   3. Intense competition in the mobility, delivery, and logistics industries, with low-cost alternatives and well-capitalized competitors.
   4. The need to lower fares, offer driver incentives, and provide consumer discounts and promotions to remain competitive in certain markets.
   5. Uber's history of significant losses and the expectation of increased operating expenses in the future, which may affect profitability.
   6. The importance of attracting and maintaining a critical mass of drivers, consumers, merchants, shippers, and carriers to keep the platform appealing.
   7. The significance of maintaining and enhancing Uber's brand and reputation, as negative publicity could harm the business.
   8. The potential impact of economic conditions and discretionary consumer spending on Uber's business.
   9. The adverse effect of increasing costs, such as fuel, food, labor, energy, and inflation, on Uber's operating results.
   10. The risk of security or privacy breaches and unauthorized access to Uber's proprietary or confidential data.
   11. The potential harm to Uber's reputation, business, and operating results from cyberattacks.
   12. The impact of climate change risks, including physical and transitional risks, on Uber's business.
   13. The commitment to climate-related initiatives that require significant effort, resources, and management time.
   14. The reliance on third parties for distributing Uber's platform and providing software, with the risk of interference or limitations.
   15. The need for additional capital to support Uber's business growth, with uncertainty about its availability on reasonable terms.
   16. The risks associated with identifying, acquiring, and integrating suitable businesses.
   17. The potential limitations and modifications to Uber's business model in certain jurisdictions.
   18. The legal and regulatory risks that could adversely impact Uber's business and future prospects.
   19. The extensive government regulation and oversight related to payment and financial services provided by Uber.
   20. The risks associated with data collection, use, transfer, disclosure, and processing, including investigations, litigation, and fines.

Context-Augmented OpenAI Agent

   21. The importance of protecting Uber's intellectual property and the risk of claims of misappropriation.
   22. The volatility and potential decline in the market price of Uber's common stock, which may not reflect operating performance.
   Please note that this is a summary of the risk factors mentioned in Uber's March 2022 10Q filing. For more detailed information, please refer to the official filing.
   context_agent.chat("What is the X and Z in September 2022?")
Use Uber 10-Q as context, use Calculator as Tool
   from llama_index.tools import BaseTool, FunctionTool
   def magic_formula(revenue: int, cost: int) -> int:
       """Runs MAGIC_FORMULA on revenue and cost."""
       return revenue - cost
   magic_tool = FunctionTool.from_defaults(fn=magic_formula, name="magic_formula")
   context_agent = ContextRetrieverOpenAIAgent.from_tools_and_retriever(
       [magic_tool], sept_index.as_retriever(similarity_top_k=3), verbose=True
   )
   response = context_agent.chat("Can you run MAGIC_FORMULA on Uber's revenue and cost?")
   Context information is below.
   ---------------------
   Three Months Ended September 30, Nine Months Ended September 30,
   2021 2022 2021 2022
   Revenue 100 % 100 % 100 % 100 %
   Costs and expenses
   Cost of revenue, exclusive of depreciation and amortization shown separately
   below 50 % 62 % 53 % 62 %
   Operations and support 10 % 7 % 11 % 8 %
   Sales and marketing 24 % 14 % 30 % 16 %
   Research and development 10 % 9 % 13 % 9 %
   General and administrative 13 % 11 % 15 % 10 %
   Depreciation and amortization 4 % 3 % 6 % 3 %
   Total costs and expenses 112 % 106 % 128 % 107 %
   Loss from operations (12)% (6)% (28)% (7)%
   Interest expense (3)% (2)% (3)% (2)%
   Other income (expense), net (38)% (6)% 16 % (34)%
   Loss before income taxes and income (loss) from equity method
   investments (52)% (14)% (16)% (43)%
   Provision for (benefit from) income taxes (2)% 1 % (3)% — %
   Income (loss) from equity method investments — % — % — % — %
   Net loss including non-controlling interests (50)% (14)% (12)% (42)%
   Less: net income (loss) attributable to non-controlling interests,
   net of tax — % — % (1)% — %
   Net loss attributable to Uber Technologies, Inc. (50)% (14)% (12)% (42)%
    Totals of percentage of revenues may not foot due to rounding.
   The following discussion and analysis is for the three and nine months ended September 30, 2022 compared to same period in 2021.
   Revenue
   Three Months Ended September 30, Nine Months Ended September 30,
   (In millions, except per centages) 2021 2022 % Change 2021 2022 % Change
   Revenue $ 4,845 $ 8,343 72 %$ 11,677 $ 23,270 99 %
   Three Months Ended September 30, 2022 Compared with the Same Period in 2021
   Revenue increased $3.5 billion, or 72%, primarily attributable to an increase in Gross Bookings of 26%, or 32% on a constant currency basis. The increase in

Context-Augmented OpenAI Agent

   Gross Bookings was primarily driven by increases in Mobility Trip volumes as the business recovers from the impacts of COVID-19 and a $1.3 billion increase in
   Freight Gross Bookings resulting primarily from the acquisition of Transplace in the fourth quarter of 2021. Additionally, during the third quarter of 2022, we saw a
   $1.1 billion increase in Mobility revenue as a result of business model changes in the UK. We also saw a $164 million increase in Delivery revenue resulting from
   an increase in certain Courier payments and incentives that are recorded in cost of revenue, exclusive of depreciation and amortization, for certain markets where
   we are primarily responsible for Delivery services and pay Couriers for services provided.
   Nine Months Ended September 30, 2022 Compared with the Same Period in 2021
   Revenue increased $11.6 billion, or 99%, primarily attributable to an increase in Gross Bookings of 31%, or 36% on a constant currency basis. The increase in
   Gross Bookings was primarily driven by increases in Mobility Trip volumes as the business recovers from the impacts of COVID-19 and a $4.4 billion increase in
   Freight Gross Bookings resulting primarily from the acquisition of Transplace in the fourth quarter of 2021. Additionally, during the first nine months of 2022, we
   saw a $2.2 billion net increase in Mobility revenue as a result of business model changes in the UK and an accrual made for the resolution of historical claims in
   the UK relating to the classification of drivers. We also saw a $751 million increase in Delivery revenue resulting from an increase in certain Courier payments and
   incentives that are recorded in cost of revenue, exclusive of depreciation and amortization, for certain markets where we are primarily responsible for
   UBER TECHNOLOGIES, INC.
   CONDENSED CONSOLIDATED STATEMENTS OF OPERATIONS
   (In millions, except share amounts which are reflected in thousands, and per share amounts)
   (Unaudited)
   Three Months Ended September  30, Nine Months Ended September  30,
   2021 2022 2021 2022
   Revenue $ 4,845 $ 8,343 $ 11,677 $ 23,270
   Costs and expenses
   Cost of revenue, exclusive of depreciation and amortization shown separately
   below 2,438 5,173 6,247 14,352
   Operations and support 475 617 1,330 1,808
   Sales and marketing 1,168 1,153 3,527 3,634
   Research and development 493 760 1,496 2,051
   General and administrative 625 908 1,705 2,391
   Depreciation and amortization 218 227 656 724
   Total costs and expenses 5,417 8,838 14,961 24,960
   Loss from operations (572) (495) (3,284) (1,690)
   Interest expense (123) (146) (353) (414)
   Other income (expense), net (1,832) (535) 1,821 (7,796)
   Loss before income taxes and income (loss) from equity method investments (2,527) (1,176) (1,816) (9,900)
   Provision for (benefit from) income taxes (101) 58 (395) (97)
   Income (loss) from equity method investments (13) 30 (28) 65
   Net loss including non-controlling interests (2,439) (1,204) (1,449) (9,738)
   Less: net income (loss) attributable to non-controlling interests, net of

Context-Augmented OpenAI Agent

   tax (15) 2 (61) (2)
   Net loss attributable to Uber Technologies, Inc. $ (2,424)$ (1,206)$ (1,388)$ (9,736)
   Net loss per share attributable to Uber Technologies, Inc. common
   stockholders:
   Basic $ (1.28)$ (0.61)$ (0.74)$ (4.96)
   Diluted $ (1.28)$ (0.61)$ (0.75)$ (4.97)
   Weighted-average shares used to compute net loss per share attributable to
   common stockholders:
   Basic 1,898,954 1,979,299 1,877,655 1,964,483
   Diluted 1,898,954 1,979,299 1,878,997 1,968,228
   The accompanying notes are an integral part of these condensed consolidated financial statements.
   5
   Components of Results of Operations
   Revenue
   We generate substantially all of our revenue from fees paid by Drivers and Merchants for use of our platform. We have concluded that we are an agent in these
   arrangements as we arrange for other parties to provide the service to the end-user. Under this model, revenue is net of Driver and Merchant earnings and Driver
   incentives. We act as an agent in these transactions by connecting consumers to Drivers and Merchants to facilitate a Trip, meal or grocery delivery service.
   During the first quarter of 2022, we modified our arrangements in certain markets and, as a result, concluded we are responsible for the provision of mobility
   services to end-users in those markets. We have determined that in these transactions, end-users are our customers and our sole performance obligation in the
   transaction is to provide transportation services to the end-user. We recognize revenue when a trip is complete. In these markets where we are responsible for
   mobility services, we present revenue from end-users on a gross basis, as we control the service provided by Drivers to end-users, while payments to Drivers in
   exchange for mobility services are recognized in cost of revenue, exclusive of depreciation and amortization.
   For additional discussion related to our revenue, see the section titled “Management’s Discussion and Analysis of Financial Condition and Results of
   Operations - Critical Accounting Estimates - Revenue Recognition,” “Note 1 - Description of Business and Summary of Significant Accounting Policies - Revenue
   Recognition,” and “Note 2 - Revenue” to our audited consolidated financial statements included in our Annual Report Form 10-K for the year ended December 31,
   2021 and Note 2 – Revenue in this Quarterly Report on Form 10-Q.
   Cost of Revenue, Exclusive of Depreciation and Amortization
   Cost of revenue, exclusive of depreciation and amortization, primarily consists of certain insurance costs related to our Mobility and Delivery offerings, credit
   card processing fees, bank fees, data center and networking expenses, mobile device and service costs, costs incurred with Carriers for Uber Freight transportation
   services, amounts related to fare chargebacks and other credit card losses as well as costs incurred for certain Mobility and Delivery transactions where we are
   primarily responsible for mobility or delivery services and pay Drivers and Couriers for services.
   We expect that cost of revenue, exclusive of depreciation and amortization, will fluctuate on an absolute dollar basis for the foreseeable future in line with Trip
   volume changes on the platform. As Trips increase or decrease, we expect related changes for insurance costs, credit card processing fees, hosting and co-located
   data center expenses, maps license fees, and other cost of revenue, exclusive of depreciation and amortization.
   Operations and Support
   Operations and support expenses primarily consist of compensation expenses, including stock-based compensation, for employees that support operations in
   cities, including the general managers, Driver operations, platform user support representatives and community managers. Also included is the cost of customer

Context-Augmented OpenAI Agent

   support, Driver background checks and the allocation of certain corporate costs.
   As our business recovers from the impacts of COVID-19 and Trip volume increases, we would expect operations and support expenses to increase on an
   absolute dollar basis for the foreseeable future, but decrease as a percentage of revenue as we become more efficient in supporting platform users.
   Sales and Marketing
   Sales and marketing expenses primarily consist of compensation costs, including stock-based compensation to sales and marketing employees, advertising
   costs, product marketing costs and discounts, loyalty programs, promotions, refunds, and credits provided to end-users who are not customers, and the allocation of
   certain corporate costs. We expense advertising and other promotional expenditures as incurred.
   As our business recovers from the impacts of COVID-19, we would anticipate sales and marketing expenses to increase on an absolute dollar basis for
   ---------------------
   Given the context information and not prior knowledge, either pick the corresponding tool or answer the function: Can you run MAGIC_FORMULA on Uber's revenue and cost?
   === Calling Function ===
   Calling function: magic_formula with args: {
     "revenue": 23270,
     "cost": 24960
   }
   Got output: -1690
   ========================
   print(response)
   The result of running MAGIC_FORMULA on Uber's revenue and cost is -1690.


OpenAI Agent + Query Engine Experimental Cookbook

In this notebook, we try out the OpenAIAgent across a variety of query
engine tools and datasets. We explore how OpenAIAgent can
compare/replace existing workflows solved by our retrievers/query
engines.
* Auto retrieval
* Joint SQL and vector search
AutoRetrieval from a Vector Database
Our existing "auto-retrieval" capabilities (in
"VectorIndexAutoRetriever") allow an LLM to infer the right query
parameters for a vector database - including both the query string and
metadata filter.
Since the OpenAI Function API can infer function parameters, we
explore its capabilities in performing auto-retrieval here.
   import pinecone
   import os
   api_key = os.environ["PINECONE_API_KEY"]
   pinecone.init(api_key=api_key, environment="us-west1-gcp")
   # dimensions are for text-embedding-ada-002
   try:
       pinecone.create_index(
           "quickstart", dimension=1536, metric="euclidean", pod_type="p1"
       )
   except Exception:
       # most likely index already exists
       pass
   pinecone_index = pinecone.Index("quickstart")
   # Optional: delete data in your pinecone index
   pinecone_index.delete(deleteAll=True, namespace="test")
   {}
   from llama_index import VectorStoreIndex, StorageContext
   from llama_index.vector_stores import PineconeVectorStore
   from llama_index.schema import TextNode
   nodes = [
       TextNode(
           text="Michael Jordan is a retired professional basketball player, widely regarded as one of the greatest basketball players of all time.",
           metadata={
               "category": "Sports",
               "country": "United States",
           },
       ),
       TextNode(
           text="Angelina Jolie is an American actress, filmmaker, and humanitarian. She has received numerous awards for her acting and is known for her philanthropic work.",
           metadata={
               "category": "Entertainment",
               "country": "United States",
           },
       ),
       TextNode(
           text="Elon Musk is a business magnate, industrial designer, and engineer. He is the founder, CEO, and lead designer of SpaceX, Tesla, Inc., Neuralink, and The Boring Company.",
           metadata={
               "category": "Business",
               "country": "United States",
           },
       ),
       TextNode(
           text="Rihanna is a Barbadian singer, actress, and businesswoman. She has achieved significant success in the music industry and is known for her versatile musical style.",
           metadata={
               "category": "Music",
               "country": "Barbados",
           },
       ),
       TextNode(
           text="Cristiano Ronaldo is a Portuguese professional footballer who is considered one of the greatest football players of all time. He has won numerous awards and set multiple records during his career.",
           metadata={
               "category": "Sports",
               "country": "Portugal",
           },
       ),
   ]
   vector_store = PineconeVectorStore(pinecone_index=pinecone_index, namespace="test")
   storage_context = StorageContext.from_defaults(vector_store=vector_store)
   index = VectorStoreIndex(nodes, storage_context=storage_context)
   Upserted vectors: 100%|███████████████████████████████████████████████████| 5/5 [00:00<00:00,  9.61it/s]
Define Function Tool
Here we define the function interface, which is passed to OpenAI to
perform auto-retrieval.
We were not able to get OpenAI to work with nested pydantic objects or
tuples as arguments, so we converted the metadata filter keys and
values into lists for the function API to work with.
   # define function tool
   from llama_index.tools import FunctionTool

   from llama_index.vector_stores.types import (
       VectorStoreInfo,
       MetadataInfo,
       ExactMatchFilter,
       MetadataFilters,
   )
   from llama_index.retrievers import VectorIndexRetriever
   from llama_index.query_engine import RetrieverQueryEngine
   from typing import List, Tuple, Any
   from pydantic import BaseModel, Field
   # hardcode top k for now
   top_k = 3
   # define vector store info describing schema of vector store
   vector_store_info = VectorStoreInfo(
       content_info="brief biography of celebrities",
       metadata_info=[
           MetadataInfo(
               name="category",
               type="str",
               description="Category of the celebrity, one of [Sports, Entertainment, Business, Music]",
           ),
           MetadataInfo(
               name="country",
               type="str",
               description="Country of the celebrity, one of [United States, Barbados, Portugal]",
           ),
       ],
   )
   # define pydantic model for auto-retrieval function
   class AutoRetrieveModel(BaseModel):
       query: str = Field(..., description="natural language query string")
       filter_key_list: List[str] = Field(
           ..., description="List of metadata filter field names"
       )
       filter_value_list: List[str] = Field(
           ...,
           description=(
               "List of metadata filter field values (corresponding to names specified in filter_key_list)"
           ),
       )
   def auto_retrieve_fn(
       query: str, filter_key_list: List[str], filter_value_list: List[str]
   ):
       """Auto retrieval function.
       Performs auto-retrieval from a vector database, and then applies a set of filters.
       """
       query = query or "Query"
       exact_match_filters = [
           ExactMatchFilter(key=k, value=v)
           for k, v in zip(filter_key_list, filter_value_list)
       ]
       retriever = VectorIndexRetriever(
           index, filters=MetadataFilters(filters=exact_match_filters), top_k=top_k
       )
       query_engine = RetrieverQueryEngine.from_args(retriever)
       response = query_engine.query(query)
       return str(response)
   description = f"""\
   Use this tool to look up biographical information about celebrities.
   The vector database schema is given below:
   {vector_store_info.json()}
   """
   auto_retrieve_tool = FunctionTool.from_defaults(
       fn=auto_retrieve_fn,
       name="celebrity_bios",
       description=description,
       fn_schema=AutoRetrieveModel,
   )
Initialize Agent
   from llama_index.agent import OpenAIAgent
   from llama_index.llms import OpenAI
   agent = OpenAIAgent.from_tools(
       [auto_retrieve_tool], llm=OpenAI(temperature=0, model="gpt-4-0613"), verbose=True
   )
   response = agent.chat("Tell me about two celebrities from the United States. ")
   print(str(response))
   === Calling Function ===
   Calling function: celebrity_bios with args: {
     "query": "celebrities",
     "filter_key_list": ["country"],
     "filter_value_list": ["United States"]
   }
   Got output:
   Celebrities in the United States who are associated with Entertainment and Sports include Angelina Jolie and Michael Jordan.
   ========================
   Angelina Jolie is an American actress, filmmaker, and humanitarian. She has received an Academy Award and three Golden Globe Awards, and has been cited as Hollywood's highest-paid actress. Jolie made her screen debut as a child alongside her father, Jon Voight, in Lookin' to Get Out (1982), and her film career began in earnest a decade later with the low-budget production Cyborg 2 (1993), followed by her first leading role in a major film, Hackers (1995).

   Michael Jordan is a retired professional basketball player from the United States. He is widely regarded as one of the greatest basketball players in history. Jordan was one of the most effectively marketed athletes of his generation and was instrumental in popularizing the NBA around the world in the 1980s and 1990s. He played 15 seasons in the NBA, winning six championships with the Chicago Bulls.
Joint Text-to-SQL and Semantic Search
This is currenty handled by our "SQLAutoVectorQueryEngine".
Let's try implementing this by giving our "OpenAIAgent" access to two
query tools: SQL and Vector
Load and Index Structured Data
We load sample structured datapoints into a SQL db and index it.
   from sqlalchemy import (
       create_engine,
       MetaData,
       Table,
       Column,
       String,
       Integer,
       select,
       column,
   )
   from llama_index import SQLDatabase, SQLStructStoreIndex
   engine = create_engine("sqlite:///:memory:", future=True)
   metadata_obj = MetaData()
   # create city SQL table
   table_name = "city_stats"
   city_stats_table = Table(
       table_name,
       metadata_obj,
       Column("city_name", String(16), primary_key=True),
       Column("population", Integer),
       Column("country", String(16), nullable=False),
   )
   metadata_obj.create_all(engine)
   # print tables
   metadata_obj.tables.keys()
   dict_keys(['city_stats'])
   from sqlalchemy import insert
   rows = [
       {"city_name": "Toronto", "population": 2930000, "country": "Canada"},
       {"city_name": "Tokyo", "population": 13960000, "country": "Japan"},
       {"city_name": "Berlin", "population": 3645000, "country": "Germany"},
   ]
   for row in rows:
       stmt = insert(city_stats_table).values(**row)
       with engine.begin() as connection:
           cursor = connection.execute(stmt)
   with engine.connect() as connection:
       cursor = connection.exec_driver_sql("SELECT * FROM city_stats")
       print(cursor.fetchall())
   [('Toronto', 2930000, 'Canada'), ('Tokyo', 13960000, 'Japan'), ('Berlin', 3645000, 'Germany')]
   sql_database = SQLDatabase(engine, include_tables=["city_stats"])
   from llama_index.indices.struct_store.sql_query import NLSQLTableQueryEngine
   query_engine = NLSQLTableQueryEngine(
       sql_database=sql_database,
       tables=["city_stats"],
   )
Load and Index Unstructured Data
We load unstructured data into a vector index backed by Pinecone
   # install wikipedia python package
   !pip install wikipedia
   Requirement already satisfied: wikipedia in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (1.4.0)
   Requirement already satisfied: requests<3.0.0,>=2.0.0 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from wikipedia) (2.28.2)
   Requirement already satisfied: beautifulsoup4 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from wikipedia) (4.12.2)
   Requirement already satisfied: charset-normalizer<4,>=2 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.1.0)
   Requirement already satisfied: idna<4,>=2.5 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.4)

   Requirement already satisfied: certifi>=2017.4.17 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2022.12.7)
   Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (1.26.15)
   Requirement already satisfied: soupsieve>1.2 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from beautifulsoup4->wikipedia) (2.4.1)
   [notice] A new release of pip available: 22.3.1 -> 23.1.2
   [notice] To update, run: pip install --upgrade pip
   from llama_index import WikipediaReader, SimpleDirectoryReader, VectorStoreIndex
   cities = ["Toronto", "Berlin", "Tokyo"]
   wiki_docs = WikipediaReader().load_data(pages=cities)
   # define pinecone index
   import pinecone
   import os
   api_key = os.environ["PINECONE_API_KEY"]
   pinecone.init(api_key=api_key, environment="us-west1-gcp")
   # dimensions are for text-embedding-ada-002
   # pinecone.create_index("quickstart", dimension=1536, metric="euclidean", pod_type="p1")
   pinecone_index = pinecone.Index("quickstart")
   # OPTIONAL: delete all
   pinecone_index.delete(deleteAll=True)
   {}
   from llama_index.node_parser import SimpleNodeParser
   from llama_index import ServiceContext
   from llama_index.storage import StorageContext
   from llama_index.vector_stores import PineconeVectorStore
   from llama_index.text_splitter import TokenTextSplitter
   from llama_index.llms import OpenAI
   # define node parser and LLM
   chunk_size = 1024
   llm = OpenAI(temperature=0, model="gpt-4")
   service_context = ServiceContext.from_defaults(chunk_size=chunk_size, llm=llm)
   text_splitter = TokenTextSplitter(chunk_size=chunk_size)
   node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)
   # define pinecone vector index
   vector_store = PineconeVectorStore(
       pinecone_index=pinecone_index, namespace="wiki_cities"
   )
   storage_context = StorageContext.from_defaults(vector_store=vector_store)
   vector_index = VectorStoreIndex([], storage_context=storage_context)
   # Insert documents into vector index
   # Each document has metadata of the city attached
   for city, wiki_doc in zip(cities, wiki_docs):
       nodes = node_parser.get_nodes_from_documents([wiki_doc])
       # add metadata to each node
       for node in nodes:
           node.metadata = {"title": city}
       vector_index.insert_nodes(nodes)
   Upserted vectors: 100%|█████████████████████████████████████████████████| 20/20 [00:00<00:00, 38.13it/s]

   Upserted vectors: 100%|████████████████████████████████████████████████| 21/21 [00:00<00:00, 101.89it/s]
   Upserted vectors: 100%|█████████████████████████████████████████████████| 13/13 [00:00<00:00, 97.91it/s]
Define Query Engines / Tools
   from llama_index.query_engine import SQLAutoVectorQueryEngine, RetrieverQueryEngine
   from llama_index.tools.query_engine import QueryEngineTool
   from llama_index.indices.vector_store import VectorIndexAutoRetriever
   from llama_index.indices.vector_store.retrievers import VectorIndexAutoRetriever
   from llama_index.vector_stores.types import MetadataInfo, VectorStoreInfo
   from llama_index.query_engine.retriever_query_engine import RetrieverQueryEngine
   vector_store_info = VectorStoreInfo(
       content_info="articles about different cities",
       metadata_info=[
           MetadataInfo(name="title", type="str", description="The name of the city"),
       ],
   )
   vector_auto_retriever = VectorIndexAutoRetriever(
       vector_index, vector_store_info=vector_store_info
   )
   retriever_query_engine = RetrieverQueryEngine.from_args(
       vector_auto_retriever, service_context=service_context
   )
   sql_tool = QueryEngineTool.from_defaults(
       query_engine=query_engine,
       name="sql_tool",
       description=(
           "Useful for translating a natural language query into a SQL query over a table containing: "
           "city_stats, containing the population/country of each city"
       ),
   )
   vector_tool = QueryEngineTool.from_defaults(
       query_engine=retriever_query_engine,
       name="vector_tool",
       description=f"Useful for answering semantic questions about different cities",
   )
Initialize Agent
   from llama_index.agent import OpenAIAgent
   from llama_index.llms import OpenAI
   agent = OpenAIAgent.from_tools(
       [sql_tool, vector_tool], llm=OpenAI(temperature=0, model="gpt-4-0613"), verbose=True
   )
   # NOTE: gpt-3.5 gives the wrong answer, but gpt-4 is able to reason over both loops
   response = agent.chat(
       "Tell me about the arts and culture of the city with the highest population"
   )
   print(str(response))
   === Calling Function ===
   Calling function: sql_tool with args: {
     "input": "SELECT city FROM city_stats ORDER BY population DESC LIMIT 1"
   }
   Got output:  The city with the highest population is Tokyo.
   ========================
   === Calling Function ===
   Calling function: vector_tool with args: {
     "input": "Tell me about the arts and culture of Tokyo"
   }
   Got output: Tokyo has a rich arts and culture scene, with many theaters for performing arts, including national and private theaters for traditional forms of Japanese drama. Noteworthy theaters are the National Noh Theatre for noh and the Kabuki-za for Kabuki. Symphony orchestras and other musical organizations perform modern and traditional music. The New National Theater Tokyo in Shibuya is the national center for the performing arts, including opera, ballet, contemporary dance, and drama. Tokyo also hosts modern Japanese and international pop and rock music at various venues, ranging from intimate clubs to internationally known areas such as the Nippon Budokan.
   Many different festivals occur throughout Tokyo, with major events including the Sannō at Hie Shrine, the Sanja at Asakusa Shrine, and the biennial Kanda Festivals. Annually on the last Saturday of July, a massive fireworks display over the Sumida River attracts over a million viewers. Once cherry blossoms bloom in spring, residents gather in Ueno Park, Inokashira Park, and the Shinjuku Gyoen National Garden for picnics under the blossoms. Harajuku, a neighborhood in Shibuya, is known internationally for its youth style, fashion, and cosplay.

   Tokyo is also renowned for its fine dining, with Michelin awarding a significant number of stars to the city's restaurants. As of 2017, 227 restaurants in Tokyo have been awarded Michelin stars, surpassing the number awarded in Paris.
   ========================
   Tokyo, the city with the highest population, has a rich arts and culture scene. It is home to many theaters for performing arts, including national and private theaters for traditional forms of Japanese drama such as Noh and Kabuki. The New National Theater Tokyo in Shibuya is the national center for the performing arts, including opera, ballet, contemporary dance, and drama.
   Tokyo also hosts modern Japanese and international pop and rock music at various venues, ranging from intimate clubs to internationally known areas such as the Nippon Budokan.
   The city is known for its festivals, with major events including the Sannō at Hie Shrine, the Sanja at Asakusa Shrine, and the biennial Kanda Festivals. Once cherry blossoms bloom in spring, residents gather in Ueno Park, Inokashira Park, and the Shinjuku Gyoen National Garden for picnics under the blossoms.
   Harajuku, a neighborhood in Shibuya, is known internationally for its youth style, fashion, and cosplay. Tokyo is also renowned for its fine dining, with Michelin awarding a significant number of stars to the city's restaurants. As of 2017, 227 restaurants in Tokyo have been awarded Michelin stars, surpassing the number awarded in Paris.
   response = agent.chat("Tell me about the history of Berlin")
   print(str(response))
   === Calling Function ===
   Calling function: vector_tool with args: {
     "input": "Tell me about the history of Berlin"
   }
   Got output: Berlin's history dates back to the 15th century when it was established as the capital of the Margraviate of Brandenburg. The Hohenzollern family ruled Berlin until 1918, first as electors of Brandenburg, then as kings of Prussia, and eventually as German emperors. In 1443, Frederick II Irontooth started the construction of a new royal palace in the twin city Berlin-Cölln, which later became the permanent residence of the Brandenburg electors of the Hohenzollerns.
   The Thirty Years' War between 1618 and 1648 devastated Berlin, with the city losing half of its population. Frederick William, known as the "Great Elector", initiated a policy of promoting immigration and religious tolerance. In 1701, the dual state of the Margraviate of Brandenburg and the Duchy of Prussia formed the Kingdom of Prussia, with Berlin as its capital. Under the rule of Frederick II, Berlin became a center of the Enlightenment.
   The Industrial Revolution in the 19th century transformed Berlin, expanding its economy and population. In 1871, Berlin became the capital of the newly founded German Empire. The early 20th century saw Berlin as a fertile ground for the German Expressionist movement. At the end of the First World War in 1918, a republic was proclaimed, and in 1920, the Greater Berlin Act incorporated dozens of suburban cities, villages, and estates around Berlin.
   ========================
   Response(response='Berlin\'s history dates back to the 15th century when it was established as the capital of the Margraviate of Brandenburg. The Hohenzollern family ruled Berlin until 1918, first as electors of Brandenburg, then as kings of Prussia, and eventually as German emperors. In 1443, Frederick II Irontooth started the construction of a new royal palace in the twin city Berlin-Cölln.\n\nThe Thirty Years\' War between 1618 and 1648 devastated Berlin, with the city losing half of its population. Frederick William, known as the "Great Elector", initiated a policy of promoting immigration and religious tolerance. In 1701, the dual state of the Margraviate of Brandenburg and the Duchy of Prussia formed the Kingdom of Prussia, with Berlin as its capital. Under the rule of Frederick II, Berlin became a center of the Enlightenment.\n\nThe Industrial Revolution in the 19th century transformed Berlin, expanding its economy and population. In 1871, Berlin became the capital of the newly founded German Empire. The early 20th century saw Berlin as a fertile ground for the German Expressionist movement. At the end of the First World War in 1918, a republic was proclaimed, and in 1920, the Greater Berlin Act incorporated dozens of suburban cities, villages, and estates around Berlin.', source_nodes=[], extra_info=None)

   response = agent.chat("Can you give me the country corresponding to each city?")
   print(str(response))
   === Calling Function ===
   Calling function: sql_tool with args: {
     "input": "SELECT city, country FROM city_stats"
   }
   Got output:  The cities Toronto, Tokyo, and Berlin are located in the countries Canada, Japan, and Germany respectively.
   ========================
   Response(response='Sure, here are the countries corresponding to each city:\n\n- Toronto is in Canada\n- Tokyo is in Japan\n- Berlin is in Germany', source_nodes=[], extra_info=None)


ReAct Agent

   from llama_index.agent import ReActAgent
   from llama_index.llms import OpenAI, ChatMessage
   from llama_index.tools import BaseTool, FunctionTool
   def multiply(a: int, b: int) -> int:
       """Multiple two integers and returns the result integer"""
       return a * b
   multiply_tool = FunctionTool.from_defaults(fn=multiply)
   def add(a: int, b: int) -> int:
       """Add two integers and returns the result integer"""
       return a + b
   add_tool = FunctionTool.from_defaults(fn=add)
gpt-3.5-turbo-0613
   llm = OpenAI(model="gpt-3.5-turbo-0613")
   agent = ReActAgent.from_tools([multiply_tool, add_tool], llm=llm, verbose=True)
   response = agent.chat("What is 20+2*4? Calculate step by step ")
   Thought: I need to use a tool to help me answer the question.
   Action: multiply
   Action Input: {'a': 2, 'b': 4}
   Observation: 8
   Thought: I have the result of the multiplication. Now I need to use the addition tool.
   Action: add
   Action Input: {'a': 20, 'b': 8}
   Observation: 28
   Response: 20 + 2 * 4 = 28

   response_gen = agent.stream_chat("What is 20+2*4? Calculate step by step")
   response_gen.print_response_stream()
   Thought: I need to use a tool to help me answer the question.
   Action: multiply
   Action Input: {'a': 2, 'b': 4}
   Observation: 8
   Thought: I need to use a tool to help me answer the question.
   Action: add
   Action Input: {'a': 20, 'b': 8}
   Observation: 28
   20 + 2 * 4 = 28
gpt-4
   llm = OpenAI(model="gpt-4")
   agent = ReActAgent.from_tools([multiply_tool, add_tool], llm=llm, verbose=True)
   response = agent.chat("What is 2+2*4")
   print(response)
   Thought: I need to use the multiply tool first to calculate 2*4, then use the add tool to add the result to 2.
   Action: multiply
   Action Input: {'a': 2, 'b': 4}
   Observation: 8
   Thought: Now I need to use the add tool to add 2 to the result of the multiplication.
   Action: add
   Action Input: {'a': 2, 'b': 8}
   Observation: 10

ReAct Agent

   Response: 10
   10
text-davinci-003
   llm = OpenAI(model="text-davinci-003")
   agent = ReActAgent.from_tools([multiply_tool, add_tool], llm=llm, verbose=True)
   response = agent.chat("What is 2+2*4")
   print(response)
   Thought: I need to use a tool to help me answer the question.
   Action: multiply
   Action Input: {'a': 2, 'b': 4}
   Observation: 8
   Response: 10
   10