Test Complex Queries over Multiple Documents (with and without Query Decomposition)

Query Decomposition: The ability to decompose a complex query into a
simpler query given the content of the index.
Use ChatGPT as the LLM model
   import logging
   # logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   # logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   # Uncomment if you want to temporarily disable logger
   logger = logging.getLogger()
   logger.disabled = True
   from llama_index import (
       VectorStoreIndex,
       SimpleKeywordTableIndex,
       SimpleDirectoryReader,
       ServiceContext,
   )
   /Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
Load Datasets
Load Wikipedia pages as well as Paul Graham's "What I Worked On" essay
   wiki_titles = [
       "Toronto",
       "Seattle",
       "San Francisco",
       "Chicago",
       "Boston",
       "Washington, D.C.",
       "Cambridge, Massachusetts",
       "Houston",
   ]
   from pathlib import Path
   import requests
   for title in wiki_titles:
       response = requests.get(
           "https://en.wikipedia.org/w/api.php",
           params={
               "action": "query",
               "format": "json",
               "titles": title,
               "prop": "extracts",
               # 'exintro': True,
               "explaintext": True,
           },
       ).json()
       page = next(iter(response["query"]["pages"].values()))
       wiki_text = page["extract"]
       data_path = Path("data")
       if not data_path.exists():
           Path.mkdir(data_path)
       with open(data_path / f"{title}.txt", "w") as fp:
           fp.write(wiki_text)
   # Load all wiki documents
   city_docs = {}
   for wiki_title in wiki_titles:
       city_docs[wiki_title] = SimpleDirectoryReader(
           input_files=[f"data/{wiki_title}.txt"]
       ).load_data()
Building the document indices
Build a vector index for the wiki pages about cities and persons, and
PG essay
   # # LLM Predictor (gpt-3.5-turbo)
   from llama_index.llms.openai import OpenAI
   chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(llm=chatgpt)
   /Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/langchain/llms/openai.py:661: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain.chat_models import ChatOpenAI`
     warnings.warn(
   # Build city document index
   city_indices = {}
   index_summaries = {}
   for wiki_title in wiki_titles:
       city_indices[wiki_title] = VectorStoreIndex.from_documents(
           city_docs[wiki_title], service_context=service_context
       )
       # set summary text for city
       index_summaries[wiki_title] = f"Wikipedia articles about {wiki_title}"
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 20744 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 16942 tokens

   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 23433 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 26082 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 18614 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 21649 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 12855 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 21844 tokens
Build Graph: Keyword Table Index on top of vector indices!
We compose a keyword table index on top of all the vector indices.
   from llama_index.indices.composability import ComposableGraph
   graph = ComposableGraph.from_indices(
       SimpleKeywordTableIndex,
       [index for _, index in city_indices.items()],
       [summary for _, summary in index_summaries.items()],
       max_keywords_per_chunk=50,
   )
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
Define Query Configs
**Query Transform**
   from llama_index.indices.query.query_transform.base import DecomposeQueryTransform
   decompose_transform = DecomposeQueryTransform(
       service_context.llm_predictor, verbose=True
   )
**Complex Query 1**
   # with query decomposition in subindices
   from llama_index.query_engine.transform_query_engine import TransformQueryEngine
   custom_query_engines = {}
   for index in city_indices.values():
       query_engine = index.as_query_engine(service_context=service_context)
       transform_metadata = {"index_summary": index.index_struct.summary}
       tranformed_query_engine = TransformQueryEngine(
           query_engine, decompose_transform, transform_metadata=transform_metadata
       )
       custom_query_engines[index.index_id] = tranformed_query_engine
   custom_query_engines[graph.root_index.index_id] = graph.root_index.as_query_engine(
       retriever_mode="simple",
       response_mode="tree_summarize",
       service_context=service_context,
   )
   query_engine_decompose = graph.as_query_engine(
       custom_query_engines=custom_query_engines,
   )
   response_chatgpt = query_engine_decompose.query(
       "Compare and contrast the airports in Seattle, Houston, and Toronto. "
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the airports in Seattle, Houston, and Toronto.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['toronto', 'airports', 'seattle', 'contrast', 'compare', 'houston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['toronto', 'seattle', 'houston']

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > Current query: Compare and contrast the airports in Seattle, Houston, and Toronto.
   > New query: What are some notable features of the Toronto Pearson International Airport?
   > Current query: Compare and contrast the airports in Seattle, Houston, and Toronto.
   > New query: What are some notable features of the Toronto Pearson International Airport?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1142 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1142 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 10 tokens
   > Current query: Compare and contrast the airports in Seattle, Houston, and Toronto.
   > New query: What is the name of the airport in Seattle?
   > Current query: Compare and contrast the airports in Seattle, Houston, and Toronto.
   > New query: What is the name of the airport in Seattle?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1773 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1773 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > Current query: Compare and contrast the airports in Seattle, Houston, and Toronto.
   > New query: What are the major airports in Houston?

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens
   > Current query: Compare and contrast the airports in Seattle, Houston, and Toronto.
   > New query: What are the major airports in Houston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1162 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1162 tokens

   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 254 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 254 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   print(str(response_chatgpt))
   Seattle has one major airport called Seattle-Tacoma International Airport, while Houston has two major airports called George Bush Intercontinental Airport and William P. Hobby Airport, as well as a third municipal airport called Ellington Airport. Toronto Pearson International Airport is Canada's busiest airport and offers limited commercial and passenger service to nearby destinations in Canada and the United States. All three cities have at least one major airport, but Houston has more options with two major airports.
   # without query decomposition in subindices
   custom_query_engines = {}
   for index in city_indices.values():
       query_engine = index.as_query_engine(service_context=service_context)
       custom_query_engines[index.index_id] = query_engine
   custom_query_engines[graph.root_index.index_id] = graph.root_index.as_query_engine(
       retriever_mode="simple",
       response_mode="tree_summarize",
       service_context=service_context,
   )
   query_engine = graph.as_query_engine(
       custom_query_engines=custom_query_engines,
   )
   response_chatgpt = query_engine.query(
       "Compare and contrast the airports in Seattle, Houston, and Toronto. "
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the airports in Seattle, Houston, and Toronto.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['toronto', 'airports', 'seattle', 'contrast', 'compare', 'houston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['toronto', 'seattle', 'houston']
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 14 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1114 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1114 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1799 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1799 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1186 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1186 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 196 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 196 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   str(response_chatgpt)
   'It is not possible to compare and contrast the airports in Seattle, Houston, and Toronto based on the given context information.'
**Complex Query 2**
   # with query decomposition
   response_chatgpt = query_engine_decompose.query(
       "Compare and contrast the sports environment of Houston and Boston. "
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the sports environment of Houston and Boston.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['environment', 'contrast', 'sports', 'compare', 'houston', 'boston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
   > Current query: Compare and contrast the sports environment of Houston and Boston.
   > New query: What sports teams are based in Houston?

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens
   > Current query: Compare and contrast the sports environment of Houston and Boston.
   > New query: What sports teams are based in Houston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1861 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1861 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 10 tokens
   > Current query: Compare and contrast the sports environment of Houston and Boston.
   > New query: What are some notable sports teams based in Boston?
   > Current query: Compare and contrast the sports environment of Houston and Boston.
   > New query: What are some notable sports teams based in Boston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1812 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1812 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 226 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 226 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   str(response_chatgpt)
   'Houston has sports teams for every major professional league except the National Hockey League, while Boston has teams for Major League Baseball, National Hockey League, National Basketball Association, National Football League, Major League Lacrosse, and Overwatch League. Both cities have a strong sports culture, but Boston has a more diverse range of professional sports teams.'
   # without query decomposition
   response_chatgpt = query_engine.query(
       "Compare and contrast the sports environment of Houston and Boston. "
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the sports environment of Houston and Boston.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['environment', 'contrast', 'sports', 'compare', 'houston', 'boston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1795 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1795 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1792 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1792 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 119 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 119 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   str(response_chatgpt)
   'Sorry, I cannot answer this question as there is no information provided about the sports environment of Houston or Boston in the given context information.'

   # with query decomposition
   response_chatgpt = query_engine_decompose.query(
       "Compare and contrast the sports environment of Houston and Boston. "
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the sports environment of Houston and Boston.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['environment', 'contrast', 'sports', 'compare', 'houston', 'boston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
   > Current query: Compare and contrast the sports environment of Houston and Boston.
   > New query: What sports teams are based in Houston?

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens
   > Current query: Compare and contrast the sports environment of Houston and Boston.
   > New query: What sports teams are based in Houston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1861 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1861 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > Current query: Compare and contrast the sports environment of Houston and Boston.
   > New query: What are some notable sports teams based in Boston?

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 10 tokens
   > Current query: Compare and contrast the sports environment of Houston and Boston.
   > New query: What are some notable sports teams based in Boston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1812 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1812 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 226 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 226 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   print(response_chatgpt)
   Houston has sports teams for every major professional league except the National Hockey League, while Boston has teams for Major League Baseball, National Hockey League, National Basketball Association, National Football League, Major League Lacrosse, and Overwatch League. Both cities have a strong sports culture, but Boston has a more diverse range of professional sports teams.

   # without query decomposition
   response_chatgpt = query_engine.query(
       "Compare and contrast the sports environment of Houston and Boston. "
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the sports environment of Houston and Boston.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['environment', 'contrast', 'sports', 'compare', 'houston', 'boston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1795 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1795 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1792 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1792 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 119 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 119 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   print(response_chatgpt)
   Sorry, I cannot answer this question as there is no information provided about the sports environment of Houston or Boston in the given context information.
**Complex Query 3**
   # with query decomposition
   response_chatgpt = query_engine_decompose.query(
       "Compare and contrast the arts and culture of Houston and Boston. "
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the arts and culture of Houston and Boston.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['arts', 'culture', 'contrast', 'compare', 'houston', 'boston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 9 tokens
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions in Houston?
   > Current query: Compare and contrast the arts and culture of Houston and Boston.

   > New query: What are some notable cultural institutions in Houston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1835 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1835 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions in Boston?

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 9 tokens
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions in Boston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1918 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1918 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 444 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 444 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   print(response_chatgpt)
   Both Houston and Boston have a variety of cultural institutions, including museums, performing arts organizations, and theaters. Some notable museums in both cities include the Museum of Fine Arts. However, Houston has a greater focus on contemporary art with institutions such as the Contemporary Arts Museum Houston and the Station Museum of Contemporary Art. Boston, on the other hand, has a unique museum in the Isabella Stewart Gardner Museum, which features a collection of art and artifacts in a recreated Venetian palace. In terms of performing arts, both cities have symphony orchestras and opera companies, but Boston also has a strong focus on contemporary classical music with groups such as the Boston Modern Orchestra Project. Overall, while both cities have a rich arts and culture scene, they differ in their specific areas of focus.
   # without query decomposition
   response_chatgpt = query_engine.query(
       "Compare and contrast the arts and culture of Houston and Boston. "
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the arts and culture of Houston and Boston.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['arts', 'culture', 'contrast', 'compare', 'houston', 'boston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 13 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1779 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1779 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1817 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1817 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 122 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 122 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   print(response_chatgpt)
   I'm sorry, but there is not enough information provided to compare and contrast the arts and culture of Houston and Boston.


Test Complex Queries over Multiple Documents (text-davinci-003 vs. ChatGPT)

Test complex queries over both text-davinci-003 and ChatGPT
   !pip install llama-index
   Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
   Collecting llama-index
     Downloading llama_index-0.4.17.tar.gz (122 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 122.8/122.8 KB 9.7 MB/s eta 0:00:00
     Preparing metadata (setup.py) ... done
   Collecting langchain
     Downloading langchain-0.0.98-py3-none-any.whl (337 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 337.8/337.8 KB 23.9 MB/s eta 0:00:00
   Collecting openai>=0.26.4
     Downloading openai-0.27.0-py3-none-any.whl (70 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 70.1/70.1 KB 4.1 MB/s eta 0:00:00
   Collecting dataclasses_json
     Downloading dataclasses_json-0.5.7-py3-none-any.whl (25 kB)
   Collecting transformers
     Downloading transformers-4.26.1-py3-none-any.whl (6.3 MB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 73.5 MB/s eta 0:00:00
   Requirement already satisfied: nltk in /usr/local/lib/python3.8/dist-packages (from llama-index) (3.7)
   Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from llama-index) (1.22.4)
   Collecting tenacity<8.2.0
     Downloading tenacity-8.1.0-py3-none-any.whl (23 kB)
   Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from llama-index) (1.3.5)
   Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from openai>=0.26.4->llama-index) (4.64.1)
   Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.8/dist-packages (from openai>=0.26.4->llama-index) (2.25.1)
   Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from openai>=0.26.4->llama-index) (3.8.4)
   Requirement already satisfied: marshmallow<4.0.0,>=3.3.0 in /usr/local/lib/python3.8/dist-packages (from dataclasses_json->llama-index) (3.19.0)

   Collecting marshmallow-enum<2.0.0,>=1.5.1
     Downloading marshmallow_enum-1.5.1-py2.py3-none-any.whl (4.2 kB)
   Collecting typing-inspect>=0.4.0
     Downloading typing_inspect-0.8.0-py3-none-any.whl (8.7 kB)
   Collecting deeplake<4.0.0,>=3.2.9
     Downloading deeplake-3.2.12.tar.gz (439 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 439.1/439.1 KB 31.3 MB/s eta 0:00:00
     Preparing metadata (setup.py) ... done
   Requirement already satisfied: PyYAML<7,>=6 in /usr/local/lib/python3.8/dist-packages (from langchain->llama-index) (6.0)
   Requirement already satisfied: SQLAlchemy<2,>=1 in /usr/local/lib/python3.8/dist-packages (from langchain->llama-index) (1.4.46)
   Requirement already satisfied: pydantic<2,>=1 in /usr/local/lib/python3.8/dist-packages (from langchain->llama-index) (1.10.5)
   Collecting aleph-alpha-client<3.0.0,>=2.15.0
     Downloading aleph_alpha_client-2.16.0-py3-none-any.whl (38 kB)
   Requirement already satisfied: joblib in /usr/local/lib/python3.8/dist-packages (from nltk->llama-index) (1.2.0)
   Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.8/dist-packages (from nltk->llama-index) (2022.6.2)
   Requirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from nltk->llama-index) (8.1.3)
   Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas->llama-index) (2.8.2)
   Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->llama-index) (2022.7.1)
   Collecting huggingface-hub<1.0,>=0.11.0
     Downloading huggingface_hub-0.12.1-py3-none-any.whl (190 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 190.3/190.3 KB 3.5 MB/s eta 0:00:00
   Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
     Downloading tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 48.5 MB/s eta 0:00:00

   Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers->llama-index) (23.0)
   Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers->llama-index) (3.9.0)
   Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->openai>=0.26.4->llama-index) (6.0.4)
   Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->openai>=0.26.4->llama-index) (22.2.0)
   Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->openai>=0.26.4->llama-index) (1.3.1)
   Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->openai>=0.26.4->llama-index) (1.3.3)
   Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->openai>=0.26.4->llama-index) (4.0.2)
   Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->openai>=0.26.4->llama-index) (3.0.1)
   Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->openai>=0.26.4->llama-index) (1.8.2)
   Requirement already satisfied: urllib3>=1.26 in /usr/local/lib/python3.8/dist-packages (from aleph-alpha-client<3.0.0,>=2.15.0->langchain->llama-index) (1.26.14)
   Collecting aiohttp-retry>=2.8.3
     Downloading aiohttp_retry-2.8.3-py3-none-any.whl (9.8 kB)
   Collecting aiodns>=3.0.0
     Downloading aiodns-3.0.0-py3-none-any.whl (5.0 kB)
   Collecting requests>=2.20
     Downloading requests-2.28.2-py3-none-any.whl (62 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 KB 4.7 MB/s eta 0:00:00
   Requirement already satisfied: pillow in /usr/local/lib/python3.8/dist-packages (from deeplake<4.0.0,>=3.2.9->langchain->llama-index) (8.4.0)
   Collecting boto3
     Downloading boto3-1.26.82-py3-none-any.whl (134 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.7/134.7 KB 6.0 MB/s eta 0:00:00

   Collecting pathos
     Downloading pathos-0.3.0-py3-none-any.whl (79 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.8/79.8 KB 8.9 MB/s eta 0:00:00
   Collecting humbug>=0.2.6
     Downloading humbug-0.2.8-py3-none-any.whl (13 kB)
   Collecting numcodecs
     Downloading numcodecs-0.11.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.7 MB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.7/6.7 MB 41.9 MB/s eta 0:00:00
   Collecting pyjwt
     Downloading PyJWT-2.6.0-py3-none-any.whl (20 kB)
   Collecting hub>=2.8.7
     Downloading hub-3.0.1-py3-none-any.whl (1.4 kB)
   Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers->llama-index) (4.5.0)
   Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.7.3->pandas->llama-index) (1.15.0)
   Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests>=2.20->openai>=0.26.4->llama-index) (2.10)
   Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests>=2.20->openai>=0.26.4->llama-index) (2022.12.7)
   Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.8/dist-packages (from SQLAlchemy<2,>=1->langchain->llama-index) (2.0.2)
   Collecting mypy-extensions>=0.3.0
     Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
   Collecting pycares>=4.0.0
     Downloading pycares-4.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 289.0/289.0 KB 19.6 MB/s eta 0:00:00
   Collecting s3transfer<0.7.0,>=0.6.0
     Downloading s3transfer-0.6.0-py3-none-any.whl (79 kB)

        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 KB 6.2 MB/s eta 0:00:00
   Collecting botocore<1.30.0,>=1.29.82
     Downloading botocore-1.29.82-py3-none-any.whl (10.5 MB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.5/10.5 MB 69.2 MB/s eta 0:00:00
   Collecting jmespath<2.0.0,>=0.7.1
     Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
   Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from numcodecs->deeplake<4.0.0,>=3.2.9->langchain->llama-index) (0.4)
   Collecting ppft>=1.7.6.6
     Downloading ppft-1.7.6.6-py3-none-any.whl (52 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.8/52.8 KB 4.1 MB/s eta 0:00:00
   Collecting pox>=0.3.2
     Downloading pox-0.3.2-py3-none-any.whl (29 kB)
   Requirement already satisfied: dill>=0.3.6 in /usr/local/lib/python3.8/dist-packages (from pathos->deeplake<4.0.0,>=3.2.9->langchain->llama-index) (0.3.6)
   Collecting multiprocess>=0.70.14
     Downloading multiprocess-0.70.14-py38-none-any.whl (132 kB)
        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.0/132.0 KB 10.0 MB/s eta 0:00:00
   Requirement already satisfied: cffi>=1.5.0 in /usr/local/lib/python3.8/dist-packages (from pycares>=4.0.0->aiodns>=3.0.0->aleph-alpha-client<3.0.0,>=2.15.0->langchain->llama-index) (1.15.1)
   Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi>=1.5.0->pycares>=4.0.0->aiodns>=3.0.0->aleph-alpha-client<3.0.0,>=2.15.0->langchain->llama-index) (2.21)
   Building wheels for collected packages: llama-index, deeplake
     Building wheel for llama-index (setup.py) ... done

     Created wheel for llama-index: filename=llama_index-0.4.17-py3-none-any.whl size=182750 sha256=67cb3c836e93d9d29a73307c2393d49392a4c8ceae94be552e0a91ca4b1d2cf1
     Stored in directory: /root/.cache/pip/wheels/15/bb/a9/de82e6a211b5f22899972226d5164f91546e6ac016bbd6c248
     Building wheel for deeplake (setup.py) ... done
     Created wheel for deeplake: filename=deeplake-3.2.12-py3-none-any.whl size=534308 sha256=b49c2dd3396d018a03f60c580ca9f15903b45507d648336b281f36605cb7950f
     Stored in directory: /root/.cache/pip/wheels/4b/1a/74/4b341aa1a16e01324c9728738ff705c049c3fa2a09e40d3d9f
   Successfully built llama-index deeplake
   Installing collected packages: tokenizers, tenacity, requests, pyjwt, ppft, pox, numcodecs, mypy-extensions, multiprocess, jmespath, typing-inspect, pycares, pathos, marshmallow-enum, humbug, huggingface-hub, botocore, transformers, s3transfer, openai, dataclasses_json, aiohttp-retry, aiodns, boto3, aleph-alpha-client, hub, deeplake, langchain, llama-index
     Attempting uninstall: tenacity
       Found existing installation: tenacity 8.2.1
       Uninstalling tenacity-8.2.1:
         Successfully uninstalled tenacity-8.2.1
     Attempting uninstall: requests
       Found existing installation: requests 2.25.1
       Uninstalling requests-2.25.1:
         Successfully uninstalled requests-2.25.1
   Successfully installed aiodns-3.0.0 aiohttp-retry-2.8.3 aleph-alpha-client-2.16.0 boto3-1.26.82 botocore-1.29.82 dataclasses_json-0.5.7 deeplake-3.2.12 hub-3.0.1 huggingface-hub-0.12.1 humbug-0.2.8 jmespath-1.0.1 langchain-0.0.98 llama-index-0.4.17 marshmallow-enum-1.5.1 multiprocess-0.70.14 mypy-extensions-1.0.0 numcodecs-0.11.0 openai-0.27.0 pathos-0.3.0 pox-0.3.2 ppft-1.7.6.6 pycares-4.3.0 pyjwt-2.6.0 requests-2.28.2 s3transfer-0.6.0 tenacity-8.1.0 tokenizers-0.13.2 transformers-4.26.1 typing-inspect-0.8.0
   # My OpenAI Key
   import os
   os.environ["OPENAI_API_KEY"] = ""
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleKeywordTableIndex,

       SummaryIndex,
       SimpleDirectoryReader,
       LLMPredictor,
       ServiceContext,
   )
   from llama_index.llms import OpenAI
   import requests
Load Datasets
Load Wikipedia pages as well as Paul Graham's "What I Worked On" essay
   wiki_titles = [
       "Toronto",
       "Seattle",
       "San Francisco",
       "Chicago",
       "Boston",
       "Washington, D.C.",
       "Cambridge, Massachusetts",
       "Houston",
   ]
   from pathlib import Path
   import requests
   for title in wiki_titles:
       response = requests.get(
           "https://en.wikipedia.org/w/api.php",
           params={
               "action": "query",
               "format": "json",
               "titles": title,
               "prop": "extracts",
               # 'exintro': True,
               "explaintext": True,
           },
       ).json()
       page = next(iter(response["query"]["pages"].values()))
       wiki_text = page["extract"]
       data_path = Path("data")
       if not data_path.exists():
           Path.mkdir(data_path)
       with open(data_path / f"{title}.txt", "w") as fp:
           fp.write(wiki_text)
   # Load all wiki documents
   city_docs = {}
   for wiki_title in wiki_titles:
       city_docs[wiki_title] = SimpleDirectoryReader(
           input_files=[f"data/{wiki_title}.txt"]
       ).load_data()
Building the document indices
Build a vector index for the wiki pages about cities and persons, and
PG essay
   # LLM Predictor (text-davinci-003)
   davinci = OpenAI(temperature=0, model="text-davinci-003")
   service_context_davinci = ServiceContext.from_defaults(llm=davinci)
   # # LLM Predictor (gpt-3.5-turbo)
   chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context_chatgpt = ServiceContext.from_defaults(llm=chatgpt)
   # Build city document index
   city_indices = {}
   for wiki_title in wiki_titles:
       city_indices[wiki_title] = VectorStoreIndex.from_documents(city_docs[wiki_title])
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 17592 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 14402 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 19954 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 22057 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 15733 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 18327 tokens

   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 10999 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 18480 tokens
Build Graph: Keyword Table Index on top of vector indices!
We compose a keyword table index on top of all the vector indices.
   # set summaries for each city
   index_summaries = {}
   for wiki_title in wiki_titles:
       # set summary text for city
       index_summaries[wiki_title] = f"Wikipedia articles about {wiki_title}"
   from llama_index.indices.composability import ComposableGraph
   graph = ComposableGraph.from_indices(
       SimpleKeywordTableIndex,
       [index for _, index in city_indices.items()],
       [summary for _, summary in index_summaries.items()],
       max_keywords_per_chunk=50,
   )
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
Compare Queries (text-davinci-003 vs. ChatGPT)
**Simple Query**
   query_engine_davinci = graph.as_query_engine(
       custom_query_engines={
           graph.root_index.index_id: graph.root_index.as_query_engine(
               retriever_mode="simple",
               service_context=service_context_davinci,
               response_mode="tree_summarize",
           )
       }
   )
   query_engine_chatgpt = graph.as_query_engine(
       custom_query_engines={
           graph.root_index.index_id: graph.root_index.as_query_engine(
               retriever_mode="simple",
               service_context=service_context_chatgpt,
               response_mode="tree_summarize",
           )
       }
   )
   query_str = "Tell me more about Boston"
   response_davinci = query_engine_davinci.query(query_str)
   response_chatgpt = query_engine_chatgpt.query(query_str)
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Tell me more about Boston
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['tell', 'boston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['boston']
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 5 tokens
   INFO:llama_index.indices.common_tree.base:> Building index from nodes: 1 chunks
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 802 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 4801 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 545 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 545 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Tell me more about Boston
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['tell', 'boston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['boston']
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 5 tokens
   INFO:llama_index.indices.common_tree.base:> Building index from nodes: 1 chunks
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 641 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 4580 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 308 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 308 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   print(response_davinci)
   Boston is the capital and largest city of the Commonwealth of Massachusetts and the cultural and financial center of the New England region of the Northeastern United States. It is one of the oldest municipalities in America, founded on the Shawmut Peninsula in 1630 by Puritan settlers from the English town of the same name. It is a center of scientific research and innovation, with nearly 5,000 startups, and is home to a number of colleges and universities, notably Harvard and MIT. It has a long seafaring tradition, and was a major port for both domestic and international trade in the 19th century. It has seen waves of immigration, with Irish, Germans, Lebanese, Syrians, French Canadians, and Russian and Polish Jews settling in the city. It was an early port of the Atlantic triangular slave trade in the New England colonies, but was soon overtaken. Boston is also known for its philanthropy, with households in the city claiming the highest average rate of philanthropy in the United States.
   print(response_chatgpt)
   Boston is a city in the New England region of the United States with a population of 675,647 as of 2020. It is known for its rich history and is considered the economic and cultural center of the region. The city has many firsts, including the first public park, first public or state school, first subway system, and first large public library in the United States. Boston is also a global pioneer in innovation and entrepreneurship, with nearly 5,000 startups. The city's economy includes finance, professional and business services, biotechnology, information technology, and government activities. Boston is a popular tourist destination, with Faneuil Hall alone drawing more than 20 million visitors per year. The city is home to many prestigious hospitals and universities, including Massachusetts General Hospital, Harvard Medical School, and Boston University.
**Complex Query 1**
   query_str = (
       "Tell me the airports in Seattle, Houston, and Toronto. "
       "If only one city is provided, return the airport information for that city. "
       "If airports for multiple cities are provided, compare and contrast the airports. "
   )
   response_davinci = query_engine_davinci.query(query_str)
   response_chatgpt = query_engine_chatgpt.query(query_str)

   print(response_davinci)
   The airports in Seattle, Houston, and Toronto are Seattle–Tacoma International Airport (IATA: SEA), George Bush Intercontinental Airport (IATA: IAH), Toronto Pearson International Airport (IATA: YYZ), and Billy Bishop Toronto City Airport (IATA: YTZ). Seattle–Tacoma International Airport is the largest airport in the Pacific Northwest region of the United States, serving over 44 million passengers annually. George Bush Intercontinental Airport is the largest airport in Houston, serving over 40 million passengers annually. Toronto Pearson International Airport is the busiest airport in Canada, serving over 50 million passengers annually. Billy Bishop Toronto City Airport is a smaller airport located on the Toronto Islands, serving over 2 million passengers annually.
   print(response_chatgpt)
   Airports in Seattle: Seattle-Tacoma International Airport.
   Airports in Houston: George Bush Intercontinental Airport, William P. Hobby Airport, and Ellington Airport.
   Airports in Toronto: Toronto Pearson International Airport, Billy Bishop Toronto City Airport, Buttonville Municipal Airport, and Downsview Airport.
   Seattle has one major airport, Seattle-Tacoma International Airport. Houston has three airports: George Bush Intercontinental Airport, William P. Hobby Airport, and Ellington Airport. Toronto has four airports: Toronto Pearson International Airport, Billy Bishop Toronto City Airport, Buttonville Municipal Airport, and Downsview Airport. Toronto has a mix of commercial and smaller airports, while Houston has a mix of commercial, military, government, and general aviation airports.
**Complex Query 2**
   query_str = (
       "Look at Houston and Boston. "
       "If only one city is provided, provide information about the sports teams for that city. "
       "If context for multiple cities are provided, compare and contrast the sports environment of the cities. "
   )
   response_davinci = query_engine_davinci.query(query_str)
   response_chatgpt = query_engine_chatgpt.query(query_str)
   print(response_davinci)
   Houston has teams for every major professional league. The Houston Astros are a Major League Baseball team that have won the World Series in 2017, 2022, and appeared in it in 2005, 2019, and 2021. The Houston Rockets are a National Basketball Association franchise based in the city since 1971, and have won two NBA Championships. The Houston Texans are a National Football League expansion team formed in 2002, and the Houston Dynamo is a Major League Soccer franchise that has been based in Houston since 2006, winning two MLS Cup titles. The Houston Dash team plays in the National Women's Soccer League, and the Houston SaberCats are a rugby team that plays in Major League Rugby.
   Boston also has teams for every major professional league. The Boston Red Sox are a Major League Baseball team that have won the World Series in 2004, 2007, 2013, and 2018. The Boston Celtics are a National Basketball Association team that have won 17 championships, most recently in 2008. The Boston Bruins are a National Hockey League team that have won six Stanley Cup championships, most recently in 2011. The New England Revolution is a Major League Soccer team that has been based in Boston since 1996. During a particularly impressive 17-year stretch from 2001 to 2018, the city's professional sports teams won twelve championships
   print(response_chatgpt)
   If only one city is provided, Houston has sports teams for every major professional league except the National Hockey League, including the Houston Astros (MLB), Houston Rockets (NBA), Houston Texans (NFL), Houston Dynamo (MLS), Houston Dash (National Women's Soccer League), and Houston SaberCats (rugby).
   If context for multiple cities are provided, Boston has teams in the four major North American men's professional sports leagues plus Major League Soccer, and has won 39 championships in these leagues. Boston is one of eight cities to have won championships in all four major American sports leagues. During a particularly impressive 17-year stretch from 2001 to 2018, the city's professional sports teams won twelve championships. The Celtics and Bruins remain competitive for titles in the century’s third decade, though the Patriots and Red Sox have fallen off from these recent glory days. In contrast, Houston has not won as many championships as Boston, but has hosted several major sports events, including the Super Bowl and World Series. Houston is also home to the first major esports team, the Houston Outlaws.

**Complex Query 3**
   query_str = (
       "Look at Houston and Boston. "
       "If only one city is provided, provide information about the arts and culture for that city. "
       "If context for multiple cities are provided, compare and contrast the arts and culture of the two cities. "
   )
   response_davinci = query_engine_davinci.query(query_str)
   response_chatgpt = query_engine_chatgpt.query(query_str)
   print(response_davinci)
   Houston and Boston both have a wide range of cultural attractions. In Houston, the Theater District is a 17-block area in the center of Downtown Houston that is home to the Bayou Place entertainment complex, restaurants, movies, plazas, and parks. The Museum District's cultural institutions and exhibits attract more than 7 million visitors a year. Notable facilities include The Museum of Fine Arts, the Houston Museum of Natural Science, the Contemporary Arts Museum Houston, the Station Museum of Contemporary Art, the Holocaust Museum Houston, the Children's Museum of Houston, and the Houston Zoo. Houston also has many annual events celebrating the diverse cultures of the city, such as the Houston Livestock Show and Rodeo, the Houston Gay Pride Parade, the Houston Greek Festival, Art Car Parade, the Houston Auto Show, the Houston International Festival, and the Bayou City Art Festival.
   In Boston, the Freedom Trail is a 2.5-mile walking tour of 16 historically significant sites in downtown Boston. The Museum of Fine Arts is one of the largest and most comprehensive art museums in the world, with more than 450,000 works of art. Boston also has many annual events celebrating the diverse cultures of the city, such as the Boston Marathon, the Boston Arts Festival
   print(response_chatgpt)
   There is no information about the arts and culture of Houston provided, but for Boston, there is a rich cultural history with a strong literary culture and a center for classical music. The city is also home to several art museums and galleries, including the Museum of Fine Arts and the Isabella Stewart Gardner Museum. The Institute of Contemporary Art is housed in a contemporary building designed by Diller Scofidio + Renfro in the Seaport District. Boston's South End Art and Design District (SoWa) and Newbury St. are both art gallery destinations.
**Complex Query 4**
   query_str = (
       "Look at Toronto and San Francisco. "
       "If only one city is provided, provide information about the demographics for that city. "
       "If context for multiple cities are provided, compare and contrast the demographics of the two cities. "
   )
   response_davinci = query_engine_davinci.query(query_str)
   response_chatgpt = query_engine_chatgpt.query(query_str)
   print(response_davinci)
   In Toronto, the population is 2,731,571 people, with a median age of 39.2 years. The racial makeup of the city is 51.5% White, 20.3% Asian, 8.6% African American, 0.8% Native American, 0.2% Pacific Islander, and 18.6% from other races. The city is also home to a large Hispanic population, making up 6.2% of the population. The three most commonly reported ethnic origins are White (46.9%), Asian (20.3%), and Black (8.6%). Christianity is the most commonly reported religion (48.4%), followed by no religion and secular perspectives (31.2%). English is the predominant language spoken by Torontonians with approximately 79% of residents having proficiency in the language, although only 43.2% of Torontonians reported English as their mother tongue.
   When comparing Toronto and San Francisco, we can see that Toronto has a larger population than San Francisco, with a median age that is slightly higher. The racial makeup of Toronto is slightly more White than San Francisco, while San Francisco has a larger Asian population. The Hispanic population is larger in San Francisco than in Toronto. Christianity is the

   print(response_chatgpt)
   Only information about Toronto is provided in the context, so demographics for Toronto can be provided. However, there is no context information about San Francisco to compare and contrast with Toronto.


Query Decomposition: The ability to decompose a complex query into a
simpler query given the content of the index.
Use OpenAI as the LLM model and embedding model.
   import logging
   import sys
   # logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   # logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   # Uncomment if you want to temporarily disable logger
   logger = logging.getLogger()
   logger.disabled = True
   from llama_index import (
       VectorStoreIndex,
       SimpleKeywordTableIndex,
       SummaryIndex,
       SimpleDirectoryReader,
       LLMPredictor,
       ServiceContext,
   )
   import requests
   /Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
Load Datasets
Load Wikipedia pages as well as Paul Graham's "What I Worked On" essay
   wiki_titles = [
       "Toronto",
       "Seattle",
       "San Francisco",
       "Chicago",
       "Boston",
       "Washington, D.C.",
       "Cambridge, Massachusetts",
       "Houston",
   ]
   from pathlib import Path
   import requests
   data_path = Path("data_wiki")
   for title in wiki_titles:
       response = requests.get(
           "https://en.wikipedia.org/w/api.php",
           params={
               "action": "query",
               "format": "json",
               "titles": title,
               "prop": "extracts",
               # 'exintro': True,
               "explaintext": True,
           },
       ).json()
       page = next(iter(response["query"]["pages"].values()))
       wiki_text = page["extract"]
       if not data_path.exists():
           Path.mkdir(data_path)
       with open(data_path / f"{title}.txt", "w") as fp:
           fp.write(wiki_text)
   # Load all wiki documents
   city_docs = {}
   all_docs = []
   for wiki_title in wiki_titles:
       city_docs[wiki_title] = SimpleDirectoryReader(
           input_files=[data_path / f"{wiki_title}.txt"]
       ).load_data()
       all_docs.extend(city_docs[wiki_title])
   # define service context
   service_context = ServiceContext.from_defaults(
       chunk_size=512,
   )
Building the document indices
Build a separate vector index for each wiki pages about cities.
We also build a "global" vector index, which ingest documents for
*all* cities.
This allows us to test different types of data structures!
   # Build index for each city document
   city_indices = {}
   index_summaries = {}
   for wiki_title in wiki_titles:
       print(f"Building index for {wiki_title}")
       city_indices[wiki_title] = VectorStoreIndex.from_documents(
           city_docs[wiki_title], service_context=service_context
       )
       # set summary text for city
       index_summaries[wiki_title] = f"Wikipedia articles about {wiki_title}"
   Building index for Toronto
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 27294 tokens
   Building index for Seattle
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 22263 tokens
   Building index for San Francisco
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens

   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 30887 tokens
   Building index for Chicago
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 34336 tokens
   Building index for Boston
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 24512 tokens
   Building index for Washington, D.C.
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 28480 tokens
   Building index for Cambridge, Massachusetts
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 17036 tokens
   Building index for Houston
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 28795 tokens
   # also setup a global vector index
   global_index = VectorStoreIndex.from_documents(
       all_docs, service_context=service_context
   )
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 213603 tokens
Creating the right structure to run compare/contrast queries
Our key goal in this notebook is to run compare/contrast queries
between different cities.
We currently have a separate vector index for every city document. We
want to setup a "graph" structure in order to route the query in the
right manner in order to retrieve the relevant text sections for each
city.
We compose a keyword table index on top of all the vector indices.
   from llama_index.indices.composability import ComposableGraph
   graph = ComposableGraph.from_indices(
       SimpleKeywordTableIndex,
       [index for _, index in city_indices.items()],
       [summary for _, summary in index_summaries.items()],
       max_keywords_per_chunk=50,
   )
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
Define Query Transformation + Query Configs
We also define a "query decomposition" transform. Since we have a
graph structure over multiple indexes, query decomposition allows us
to break a complex question into a simpler one over a given index.
This works well in comparing/contrasting different cities because it
allows us to ask questions specific to each city.
**Query Transform**
   from llama_index.indices.query.query_transform.base import DecomposeQueryTransform
   decompose_transform = DecomposeQueryTransform(verbose=True)
   %load_ext autoreload
   %autoreload 2
   The autoreload extension is already loaded. To reload it, use:
     %reload_ext autoreload
   from llama_index.query_engine.transform_query_engine import TransformQueryEngine
   custom_query_engines = {}
   for index in city_indices.values():
       query_engine = index.as_query_engine(service_context=service_context)

       query_engine = TransformQueryEngine(
           query_engine,
           query_transform=decompose_transform,
           transform_extra_info={"index_summary": index.index_struct.summary},
       )
       custom_query_engines[index.index_id] = query_engine
   custom_query_engines[graph.root_id] = graph.root_index.as_query_engine(
       retriever_mode="simple",
       response_mode="tree_summarize",
       service_context=service_context,
   )
Let's Run Some Queries!
We run queries over the graphs and analyze the results.
We also compare results against the baseline global vector index. In
the majority of cases the global vector index provides insufficient
answers.
**Complex Query 1**
   # with query decomposition in subindices
   query_engine = graph.as_query_engine(custom_query_engines=custom_query_engines)
   query_str = "Compare and contrast the demographics in Seattle, Houston, and Toronto. "
   response = query_engine.query(query_str)
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the demographics in Seattle, Houston, and Toronto.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['demographics', 'seattle', 'toronto', 'compare', 'contrast', 'houston']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['seattle', 'toronto', 'houston']
   > Current query: Compare and contrast the demographics in Seattle, Houston, and Toronto.
   > New query:  What is the population of Seattle?

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 7 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1375 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1375 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 7 tokens
   > Current query: Compare and contrast the demographics in Seattle, Houston, and Toronto.
   > New query:  What is the population of Toronto?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1303 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1303 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > Current query: Compare and contrast the demographics in Seattle, Houston, and Toronto.
   > New query:  What is the population of Houston?

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 7 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1401 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1401 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1681 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1681 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   print(str(response))
   Seattle, Houston, and Toronto all have diverse populations, with immigrants making up a significant portion of the population in each city. However, the countries of origin for the immigrants vary between the cities. In Seattle, the top countries of origin for immigrants are Mexico, India, China, Philippines, and Vietnam. In Houston, the top countries of origin for immigrants are Mexico, India, El Salvador, Honduras, and Guatemala. In Toronto, the top countries of origin for immigrants are Philippines, China, India, Sri Lanka, and Jamaica. Additionally, the median age of the population varies between the cities. In Seattle, the median age is 37.2, in Houston it is 33.4, and in Toronto it is 39.2. Furthermore, the gender population also varies between the cities. In Seattle, the gender population is 48.2% male and 51.8% female, in Houston it is 48.3% male and 51.7% female, and in Toronto it is 48% male and 52% female. In 2016, the three most commonly reported ethnic origins overall were Chinese (332,830 or 12.5 per cent), South Asian (323,810 or 11.9 per cent), and Black (308,345 or 11.3
   query_engine = global_index.as_query_engine(
       similarity_top_k=3, response_mode="tree_summarize"
   )
   response = query_engine.query(query_str)
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 14 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 3549 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 3549 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   # NOTE: the global vector index seems to provide the right results....
   # BUT see below!
   print(str(response))
   Seattle is a major U.S. city located in the Pacific Northwest region of the United States. It has a population of over 730,000 people and is known for its high percentage of college and university graduates. Of the city's population over the age of 25, 53.8% hold a bachelor's degree or higher, and 91.9% have a high school diploma or equivalent. Seattle is also home to the University of Washington, as well as a number of smaller private universities such as Seattle Pacific University, a Jesuit Catholic institution, and Seattle University, a Free Methodist institution. The Seattle Colleges District operates three colleges: North Seattle College, Seattle Central College, and South Seattle College. According to a 2006 study by UCLA, 12.9% of city residents polled identified as gay, lesbian, or bisexual. This was the second-highest proportion of any major U.S. city, behind San Francisco. Seattle's economy is driven by a mix of older industrial companies and "new economy" internet and technology companies, as well as service, design, and clean technology companies. It is estimated that King County has 8,000 homeless people on any given night, and many of those live in Seattle. In recent years, the city has experienced steady population

   # NOTE: there's hallucination! the sources only reference Toronto
   print(response.source_nodes[0].source_text)
   print(response.source_nodes[1].source_text)
   Tiffany Washington, and Kendee Yamaguchi.
   == Education ==
   Of the city's population over the age of 25, 53.8% (vs. a national average of 27.4%) hold a bachelor's degree or higher, and 91.9% (vs. 84.5% nationally) have a high school diploma or equivalent. A 2008 United States Census Bureau survey showed that Seattle had the highest percentage of college and university graduates of any major U.S. city. The city was listed as the most literate of the country's 69 largest cities in 2005 and 2006, the second most literate in 2007 and the most literate in 2008 in studies conducted by Central Connecticut State University.Seattle Public Schools is the school district for the vast majority of the city. That school district desegregated without a court order but continue to struggle to achieve racial balance in a somewhat ethnically divided city (the south part of town having more ethnic minorities than the north). In 2007, Seattle's racial tie-breaking system was struck down by the United States Supreme Court, but the ruling left the door open for desegregation formulae based on other indicators (e.g., income or socioeconomic class). A very small portion of the city is within the Highline School District.The public school system is supplemented by a moderate number of private schools: Five of the private high schools are Catholic, one is Lutheran, and six are secular.Seattle is home to the University of Washington, as well as the institution's professional and continuing education unit, the University of Washington Educational Outreach. The 2017 U.S. News & World Report ranked the University of Washington at No. 11 in the world. The UW receives more federal research and development funding than any public institution. Over the last 10 years, it has also produced more Peace Corps volunteers than any other U.S. university. Seattle also has a number of smaller private universities including Seattle University and Seattle Pacific University, the former a Jesuit Catholic institution, the latter a Free Methodist institution. The Seattle Colleges District operates three colleges: North Seattle College, Seattle Central College, and South Seattle College. Universities aimed at the
   bisexual, and transgender community. According to a 2006 study by UCLA, 12.9% of city residents polled identified as gay, lesbian, or bisexual. This was the second-highest proportion of any major U.S. city, behind San Francisco. Greater Seattle also ranked second among major U.S. metropolitan areas, with 6.5% of the population identifying as gay, lesbian, or bisexual. According to 2012 estimates from the United States Census Bureau, Seattle has the highest percentage of same-sex households in the United States, at 2.6 percent, surpassing San Francisco (2.5 percent). The Capitol Hill district has historically been the center of LGBT culture in Seattle.
   == Economy ==
   Seattle's economy is driven by a mix of older industrial companies and "new economy" internet and technology companies, as well as service, design, and clean technology companies. The city's gross metropolitan product (GMP) was $231 billion in 2010, making it the 11th largest metropolitan economy in the United States. The Port of Seattle, which also operates Seattle–Tacoma International Airport, is a major gateway for trade with Asia and cruises to Alaska. It also is the 8th largest port in the United States when measured by container capacity. Its maritime cargo operations merged with the Port of Tacoma in 2015 to form the Northwest Seaport Alliance. Although it was affected by the Great Recession, Seattle has retained a comparatively strong economy, and is noted for start-up businesses, especially in green building and clean technologies. In February 2010, the city government committed Seattle to become North America's first "climate neutral" city, with a goal of reaching zero net per capita greenhouse gas emissions by 2030.Large companies continue to dominate the business landscape. Seven companies on Fortune 500's 2022 list of the United States' largest companies (based on total revenue) are headquartered in Seattle: Internet retailer Amazon (#2), coffee chain Starbucks (#120), freight forwarder Expeditors International of Washington (#225), department store Nordstrom (#245), forest products company Weyerhaeuser (#354), online travel company

   /Users/suo/dev/llama_index/llama_index/data_structs/node.py:176: UserWarning: .source_text is deprecated, use .node.get_text() instead
     warnings.warn(".source_text is deprecated, use .node.get_text() instead")
**Complex Query 2**
   # with query decomposition
   query_str = "What are the basketball teams in Houston and Boston?"
   query_engine = graph.as_query_engine(custom_query_engines=custom_query_engines)
   response = query_engine.query(query_str)
   print(str(response))
   query_engine = global_index.as_query_engine(
       similarity_top_k=2, response_mode="tree_summarize"
   )
   response = query_engine.query(query_str)
   print(str(response))
**Complex Query 3**
   # with query decomposition
   query_str = "Compare and contrast the climate of Houston and Boston "
   query_engine = graph.as_query_engine(custom_query_engines=custom_query_engines)
   response = query_engine.query(query_str)
   print(response)
   query_engine = global_index.as_query_engine(
       similarity_top_k=2, response_mode="tree_summarize"
   )
   response = query_engine.query(query_str)
   print(str(response))


Github Issue Analysis

Setup
To use the github repo issue loader, you need to set your github token
in the environment.
See here for how to get a github token.See llama-hub for more details
about the loader.
   import os
   os.environ["GITHUB_TOKEN"] = "<your github token>"
Load Github Issue tickets
   import os
   from llama_hub.github_repo_issues import (
       GitHubRepositoryIssuesReader,
       GitHubIssuesClient,
   )
   github_client = GitHubIssuesClient()
   loader = GitHubRepositoryIssuesReader(
       github_client,
       owner="jerryjliu",
       repo="llama_index",
       verbose=True,
   )
   docs = loader.load_data()
   Found 100 issues in the repo page 1
   Resulted in 100 documents
   Found 100 issues in the repo page 2
   Resulted in 200 documents
   Found 100 issues in the repo page 3
   Resulted in 300 documents
   Found 100 issues in the repo page 4
   Resulted in 400 documents
   Found 4 issues in the repo page 5
   Resulted in 404 documents
   No more issues found, stopping
Quick inspection
   docs[10].text
   "feat(context length): QnA Summarization as a relevant information extractor\n### Feature Description\r\n\r\nSummarizer can help in cases where the information is evenly distributed in the document i.e. a large amount of context is required but the language is verbose or there are many irrelevant details. Summarization specific to the query can help.\r\n\r\nEither cheap local model or even LLM are options; the latter for reducing latency due to large context window in RAG. \r\n\r\nAnother place where it helps is that percentile and top_k don't account for variable information density. (However, this may be solved with inter-node sub-node reranking). \r\n"
   docs[10].metadata
   {'state': 'open',
    'created_at': '2023-07-13T11:16:30Z',
    'url': 'https://api.github.com/repos/jerryjliu/llama_index/issues/6889',
    'source': 'https://github.com/jerryjliu/llama_index/issues/6889'}
Extract themes
   %load_ext autoreload
   %autoreload 2
   The autoreload extension is already loaded. To reload it, use:
     %reload_ext autoreload
   from pydantic import BaseModel
   from typing import List
   from tqdm.asyncio import asyncio
   from llama_index.program import OpenAIPydanticProgram
   from llama_index.llms import OpenAI
   from llama_index.async_utils import batch_gather
   prompt_template_str = """\
   Here is a Github Issue ticket.
   {ticket}
   Please extract central themes and output a list of tags.\
   """
   class TagList(BaseModel):
       """A list of tags corresponding to central themes of an issue."""
       tags: List[str]
   program = OpenAIPydanticProgram.from_defaults(
       prompt_template_str=prompt_template_str,
       output_cls=TagList,
   )
   tasks = [program.acall(ticket=doc) for doc in docs]
   output = await batch_gather(tasks, batch_size=10, verbose=True)
[Optional] Save/Load Extracted Themes
   import pickle
   with open("github_issue_analysis_data.pkl", "wb") as f:
       pickle.dump(tag_lists, f)
   with open("github_issue_analysis_data.pkl", "rb") as f:
       tag_lists = pickle.load(f)
       print(f"Loaded tag lists for {len(tag_lists)} tickets")
Summarize Themes
Build prompt
   prompt = """
   Here is a list of central themes (in the form of tags) extracted from a list of Github Issue tickets.

Github Issue Analysis

   Tags for each ticket is separated by 2 newlines.
   {tag_lists_str}
   Please summarize the key takeaways and what we should prioritize to fix.
   """
   tag_lists_str = "\n\n".join([str(tag_list) for tag_list in tag_lists])
   prompt = prompt.format(tag_lists_str=tag_lists_str)
Summarize with GPT-4
   from llama_index.llms import OpenAI
   response = OpenAI(model="gpt-4").stream_complete(prompt)
   for r in response:
       print(r.delta, end="")
   1. Bug Fixes: There are numerous bugs reported across different components such as 'Updating/Refreshing documents', 'Supabase Vector Store', 'Parsing', 'Qdrant', 'LLM event', 'Service context', 'Chroma db', 'Markdown Reader', 'Search_params', 'Index_params', 'MilvusVectorStore', 'SentenceSplitter', 'Embedding timeouts', 'PGVectorStore', 'NotionPageReader', 'VectorIndexRetriever', 'Knowledge Graph', 'LLM content', and 'Query engine'. These issues need to be prioritized and resolved to ensure smooth functioning of the system.
   2. Feature Requests: There are several feature requests like 'QnA Summarization', 'BEIR evaluation', 'Cross-Node Ranking', 'Node content', 'PruningMode', 'RelevanceMode', 'Local-model defaults', 'Dynamically selecting from multiple prompts', 'Human-In-The-Loop Multistep Query', 'Explore Tree-of-Thought', 'Postprocessing', 'Relevant Section Extraction', 'Original Source Reconstruction', 'Varied Latency in Retrieval', and 'MLFlow'. These features can enhance the capabilities of the system and should be considered for future development.
   3. Code Refactoring and Testing: There are mentions of code refactoring, testing, and code review. This indicates a need for improving code quality and ensuring robustness through comprehensive testing.
   4. Documentation: There are several mentions of documentation updates, indicating a need for better documentation to help users understand and use the system effectively.
   5. Integration: There are mentions of integration with other systems like 'BEIR', 'Langflow', 'Hugging Face', 'OpenAI', 'DynamoDB', and 'CometML'. This suggests a need for better interoperability with other systems.
   6. Performance and Efficiency: There are mentions of 'Parallelize sync APIs', 'Average query time', 'Efficiency', 'Upgrade', and 'Execution Plan'. This indicates a need for improving the performance and efficiency of the system.
   7. User Experience (UX): There are mentions of 'UX', 'Varied Latency in Retrieval', and 'Human-In-The-Loop Multistep Query'. This suggests a need for improving the user experience.
   8. Error Handling: There are several mentions of error handling, indicating a need for better error handling mechanisms to ensure system robustness.
   9. Authentication: There are mentions of 'authentication' and 'API key', indicating a need for secure access mechanisms.
   10. Multilingual Support: There is a mention of 'LLM中文应用交流微信群', indicating a need for multilingual support.


10Q Analysis

In this demo, we explore answering complex queries by decomposing them
into simpler sub-queries.
   import nest_asyncio
   nest_asyncio.apply()
   from llama_index import SimpleDirectoryReader, ServiceContext, VectorStoreIndex
   from llama_index.response.pprint_utils import pprint_response
   from llama_index.llms import OpenAI
   from llama_index.tools import QueryEngineTool, ToolMetadata
   from llama_index.query_engine import SubQuestionQueryEngine
Configure LLM service
   llm = OpenAI(temperature=0, model="text-davinci-003", max_tokens=-1)
   service_context = ServiceContext.from_defaults(llm=llm)
Load data
   march_2022 = SimpleDirectoryReader(
       input_files=["../data/10q/uber_10q_march_2022.pdf"]
   ).load_data()
   june_2022 = SimpleDirectoryReader(
       input_files=["../data/10q/uber_10q_june_2022.pdf"]
   ).load_data()
   sept_2022 = SimpleDirectoryReader(
       input_files=["../data/10q/uber_10q_sept_2022.pdf"]
   ).load_data()
Build indices
   march_index = VectorStoreIndex.from_documents(march_2022)
   june_index = VectorStoreIndex.from_documents(june_2022)
   sept_index = VectorStoreIndex.from_documents(sept_2022)
Build query engines
   march_engine = march_index.as_query_engine(similarity_top_k=3)
   june_engine = june_index.as_query_engine(similarity_top_k=3)
   sept_engine = sept_index.as_query_engine(similarity_top_k=3)
   query_engine_tools = [
       QueryEngineTool(
           query_engine=sept_engine,
           metadata=ToolMetadata(
               name="sept_22",
               description="Provides information about Uber quarterly financials ending September 2022",
           ),
       ),
       QueryEngineTool(
           query_engine=june_engine,
           metadata=ToolMetadata(
               name="june_22",
               description="Provides information about Uber quarterly financials ending June 2022",
           ),
       ),
       QueryEngineTool(
           query_engine=march_engine,
           metadata=ToolMetadata(
               name="march_22",
               description="Provides information about Uber quarterly financials ending March 2022",
           ),
       ),
   ]
   s_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools)
Run queries
   response = s_engine.query(
       "Analyze Uber revenue growth over the latest two quarter filings"
   )
   Generated 2 sub questions.
   [sept_22] Q: What is the revenue growth of Uber for the quarter ending September 2022
   [sept_22] A: compared to the same period in 2021?
   The revenue growth of Uber for the quarter ending September 2022 compared to the same period in 2021 is 72%.
   [june_22] Q: What is the revenue growth of Uber for the quarter ending June 2022
   [june_22] A: compared to the same period in 2021?
   The revenue growth of Uber for the quarter ending June 2022 compared to the same period in 2021 is 105%.

   print(response)
   Uber's revenue growth over the latest two quarter filings has been strong, with a 72% increase for the quarter ending September 2022 compared to the same period in 2021, and a 105% increase for the quarter ending June 2022 compared to the same period in 2021.

10Q Analysis

   response = s_engine.query("Analyze change in macro environment over the 3 quarters")
   Generated 3 sub questions.
   [sept_22] Q: What is the macro environment in September 2022
   [sept_22] A:
   The macro environment in September 2022 is one of recovery from the impacts of the COVID-19 pandemic, with increases in Mobility Trip volumes, a $1.3 billion increase in Freight Gross Bookings resulting from the acquisition of Transplace, a $1.1 billion increase in Mobility revenue due to business model changes in the UK, and a $164 million increase in Delivery revenue due to an increase in certain Courier payments and incentives. Additionally, there was a $2.2 billion net increase in Mobility revenue due to business model changes in the UK and an accrual made for the resolution of historical claims in the UK relating to the classification of drivers, and a $751 million increase in Delivery revenue due to an increase in certain Courier payments and incentives.
   [june_22] Q: What is the macro environment in June 2022
   [june_22] A:
   In June 2022, the macro environment is characterized by the continued impact of COVID-19 restrictions on global demand, the adoption of new accounting standards, and the potential for shifts in consumer travel patterns due to health concerns.
   [march_22] Q: What is the macro environment in March 2022
   [march_22] A:
   The macro environment in March 2022 is uncertain, as the effects of the COVID-19 pandemic and the actions taken to mitigate it are still being felt. Travel restrictions, business restrictions, school closures, and limitations on social or public gatherings may still be in place in some regions, and the demand for services may still be affected.

   print(response)
   The macro environment has seen a significant change over the three quarters from March 2022 to September 2022. In March 2022, the macro environment was uncertain due to the effects of the COVID-19 pandemic and the actions taken to mitigate it. By June 2022, the macro environment was characterized by the continued impact of COVID-19 restrictions on global demand, the adoption of new accounting standards, and the potential for shifts in consumer travel patterns due to health concerns. By September 2022, the macro environment had shifted to one of recovery from the impacts of the COVID-19 pandemic, with increases in Mobility Trip volumes, a $1.3 billion increase in Freight Gross Bookings resulting from the acquisition of Transplace, a $1.1 billion increase in Mobility revenue due to business model changes in the UK, and a $164 million increase in Delivery revenue due to an increase in certain Courier payments and incentives. Additionally, there was a $2.2 billion net increase in Mobility revenue due to business model changes in the UK and an accrual made for the resolution of historical claims in the UK relating to the classification of drivers, and a $751 million increase in Delivery revenue due to an increase in certain Courier payments and incentives.
   response = s_engine.query("How much cash did Uber have in sept 2022")
   Generated 1 sub questions.
   [sept_22] Q: How much cash did Uber have in September 2022
   [sept_22] A:

10Q Analysis

   Uber had $4,865 million in cash in September 2022.

   print(response)
   Uber had $4,865 million in cash in September 2022.