Self Correcting Query Engines - Evaluation & Retry

In this notebook, we showcase several advanced, self-correcting query
engines.They leverage the latest LLM's ability to evaluate its own
output, and then self-correct to give better responses.
   # Uncomment to add your OpenAI API key
   # import os
   # os.environ['OPENAI_API_KEY'] = "INSERT OPENAI KEY"
   # Uncomment for debug level logging
   # import logging
   # import sys
   # logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
   # logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
Setup
First we ingest the document.
   from llama_index.indices.vector_store.base import VectorStoreIndex
   from llama_index.readers.file.base import SimpleDirectoryReader
   # Needed for running async functions in Jupyter Notebook
   import nest_asyncio
   nest_asyncio.apply()
   documents = SimpleDirectoryReader("../data/paul_graham/").load_data()
   index = VectorStoreIndex.from_documents(documents)
   query = "What did the author do growing up?"
Let's what the response from the default query engine looks like
   base_query_engine = index.as_query_engine()
   response = base_query_engine.query(query)
   print(response)
   The author worked on writing and programming outside of school before college. They wrote short stories and tried writing programs on an IBM 1401 computer using an early version of Fortran. They later got a microcomputer and started programming on it, writing simple games and a word processor. They also mentioned their interest in philosophy and AI.
Retry Query Engine
The retry query engine uses an evaluator to improve the response from
a base query engine.
It does the following:
1. first queries the base query engine, then
2. use the evaluator to decided if the response passes.
3. If the response passes, then return response,
4. Otherwise, transform the original query with the evaluation result
   (query, response, and feedback) into a new query,
5. Repeat up to max_retries
   from llama_index.query_engine import RetryQueryEngine
   from llama_index.evaluation import RelevancyEvaluator
   query_response_evaluator = RelevancyEvaluator()
   retry_query_engine = RetryQueryEngine(base_query_engine, query_response_evaluator)
   retry_response = retry_query_engine.query(query)
   print(retry_response)
   The author worked on writing and programming outside of school before college. They wrote short stories and tried writing programs on an IBM 1401 computer using an early version of Fortran. They later got a microcomputer, a TRS-80, and started programming more extensively, including writing simple games and a word processor.
Retry Source Query Engine
The Source Retry modifies the query source nodes by filtering the
existing source nodes for the query based on llm node evaluation.
   from llama_index.query_engine import RetrySourceQueryEngine
   retry_source_query_engine = RetrySourceQueryEngine(
       base_query_engine, query_response_evaluator
   )
   retry_source_response = retry_source_query_engine.query(query)
   print(retry_source_response)
   The author worked on writing and programming outside of school before college. They wrote short stories and tried writing programs on an IBM 1401 computer using an early version of Fortran. They later got a microcomputer and started programming on it, writing simple games and a word processor. They also mentioned their interest in philosophy and AI.
Retry Guideline Query Engine
This module tries to use guidelines to direct the evaluator's
behavior. You can customize your own guidelines.
   from llama_index.evaluation.guideline import GuidelineEvaluator, DEFAULT_GUIDELINES
   from llama_index.response.schema import Response
   from llama_index.indices.query.query_transform.feedback_transform import (
       FeedbackQueryTransformation,
   )
   from llama_index.query_engine.retry_query_engine import (
       RetryGuidelineQueryEngine,
   )
   # Guideline eval
   guideline_eval = GuidelineEvaluator(

       guidelines=DEFAULT_GUIDELINES + "\nThe response should not be overly long.\n"
       "The response should try to summarize where possible.\n"
   )  # just for example
Let's look like what happens under the hood.
   typed_response = response if isinstance(response, Response) else response.get_response()
   eval = guideline_eval.evaluate_response(query, typed_response)
   print(f"Guideline eval evaluation result: {eval.feedback}")
   feedback_query_transform = FeedbackQueryTransformation(resynthesize_query=True)
   transformed_query = feedback_query_transform.run(query, {"evaluation": eval})
   print(f"Transformed query: {transformed_query.query_str}")
   Guideline eval evaluation result: The response partially answers the query but lacks specific statistics or numbers. It provides some details about the author's activities growing up, such as writing short stories and programming on different computers, but it could be more concise and focused. Additionally, the response does not mention any statistics or numbers to support the author's experiences.
   Transformed query: Here is a previous bad answer.
   The author worked on writing and programming outside of school before college. They wrote short stories and tried writing programs on an IBM 1401 computer using an early version of Fortran. They later got a microcomputer and started programming on it, writing simple games and a word processor. They also mentioned their interest in philosophy and AI.
   Here is some feedback from the evaluator about the response given.
   The response partially answers the query but lacks specific statistics or numbers. It provides some details about the author's activities growing up, such as writing short stories and programming on different computers, but it could be more concise and focused. Additionally, the response does not mention any statistics or numbers to support the author's experiences.
   Now answer the question.
   What were the author's activities and interests during their childhood and adolescence?
Now let's run the full query engine
   retry_guideline_query_engine = RetryGuidelineQueryEngine(
       base_query_engine, guideline_eval, resynthesize_query=True
   )
   retry_guideline_response = retry_guideline_query_engine.query(query)
   print(retry_guideline_response)
   During their childhood and adolescence, the author worked on writing short stories and programming. They mentioned that their short stories were not very good, lacking plot but focusing on characters with strong feelings. In terms of programming, they tried writing programs on the IBM 1401 computer in 9th grade using an early version of Fortran. However, they mentioned being puzzled by the 1401 and not being able to do much with it due to the limited input options. They also mentioned getting a microcomputer, a TRS-80, and starting to write simple games, a program to predict rocket heights, and a word processor.


Correctness Evaluator

This notebook uses the "CorrectnessEvaluator" to evaluate the
relevance and correctness of a generated answer against a reference
answer.
   from llama_index.evaluation import CorrectnessEvaluator
   from llama_index.llms import OpenAI
   from llama_index import ServiceContext
   service_context = ServiceContext.from_defaults(llm=OpenAI("gpt-4"))
   evaluator = CorrectnessEvaluator(service_context=service_context)
   query = (
       "Can you explain the theory of relativity proposed by Albert Einstein in detail?"
   )
   reference = """
   Certainly! Albert Einstein's theory of relativity consists of two main components: special relativity and general relativity. Special relativity, published in 1905, introduced the concept that the laws of physics are the same for all non-accelerating observers and that the speed of light in a vacuum is a constant, regardless of the motion of the source or observer. It also gave rise to the famous equation E=mc², which relates energy (E) and mass (m).
   General relativity, published in 1915, extended these ideas to include the effects of gravity. According to general relativity, gravity is not a force between masses, as described by Newton's theory of gravity, but rather the result of the warping of space and time by mass and energy. Massive objects, such as planets and stars, cause a curvature in spacetime, and smaller objects follow curved paths in response to this curvature. This concept is often illustrated using the analogy of a heavy ball placed on a rubber sheet, causing it to create a depression that other objects (representing smaller masses) naturally move towards.
   In essence, general relativity provided a new understanding of gravity, explaining phenomena like the bending of light by gravity (gravitational lensing) and the precession of the orbit of Mercury. It has been confirmed through numerous experiments and observations and has become a fundamental theory in modern physics.
   """
   response = """
   Certainly! Albert Einstein's theory of relativity consists of two main components: special relativity and general relativity. Special relativity, published in 1905, introduced the concept that the laws of physics are the same for all non-accelerating observers and that the speed of light in a vacuum is a constant, regardless of the motion of the source or observer. It also gave rise to the famous equation E=mc², which relates energy (E) and mass (m).
   However, general relativity, published in 1915, extended these ideas to include the effects of magnetism. According to general relativity, gravity is not a force between masses but rather the result of the warping of space and time by magnetic fields generated by massive objects. Massive objects, such as planets and stars, create magnetic fields that cause a curvature in spacetime, and smaller objects follow curved paths in response to this magnetic curvature. This concept is often illustrated using the analogy of a heavy ball placed on a rubber sheet with magnets underneath, causing it to create a depression that other objects (representing smaller masses) naturally move towards due to magnetic attraction.
   """
   result = evaluator.evaluate(
       query=query,
       response=response,
       reference=reference,
   )
   result.score
   2.5
   result.feedback
   'The generated answer is relevant to the user query as it attempts to explain the theory of relativity proposed by Albert Einstein. However, it contains significant mistakes. The explanation of general relativity is incorrect. General relativity is about the warping of space and time by mass and energy, not magnetic fields. The analogy used in the generated answer is also incorrect as it introduces magnets, which are not part of the original analogy or the theory of general relativity. These errors significantly affect the correctness of the information provided.'


Embedding Similarity Evaluator

This notebook shows the "SemanticSimilarityEvaluator", which evaluates
the quality of a question answering system via semantic similarity.
Concretely, it calculates the similarity score between embeddings of
the generated answer and the reference answer.
   from llama_index.evaluation import SemanticSimilarityEvaluator
   evaluator = SemanticSimilarityEvaluator()
   # This evaluator only uses `response` and `reference`, passing in query does not influence the evaluation
   # query = 'What is the color of the sky'
   response = "The sky is typically blue"
   reference = """The color of the sky can vary depending on several factors, including time of day, weather conditions, and location.
   During the day, when the sun is in the sky, the sky often appears blue.
   This is because of a phenomenon called Rayleigh scattering, where molecules and particles in the Earth's atmosphere scatter sunlight in all directions, and blue light is scattered more than other colors because it travels as shorter, smaller waves.
   This is why we perceive the sky as blue on a clear day.
   """
   result = await evaluator.aevaluate(
       response=response,
       reference=reference,
   )
   print("Score: ", result.score)
   print("Passing: ", result.passing)  # default similarity threshold is 0.8
   Score:  0.874911773340899
   Passing:  True
   response = "Sorry, I do not have sufficient context to answer this question."
   reference = """The color of the sky can vary depending on several factors, including time of day, weather conditions, and location.
   During the day, when the sun is in the sky, the sky often appears blue.
   This is because of a phenomenon called Rayleigh scattering, where molecules and particles in the Earth's atmosphere scatter sunlight in all directions, and blue light is scattered more than other colors because it travels as shorter, smaller waves.
   This is why we perceive the sky as blue on a clear day.
   """
   result = await evaluator.aevaluate(
       response=response,
       reference=reference,
   )
   print("Score: ", result.score)
   print("Passing: ", result.passing)  # default similarity threshold is 0.8
   Score:  0.7221738929165528
   Passing:  False
Customization
   from llama_index.evaluation import SemanticSimilarityEvaluator
   from llama_index import ServiceContext
   from llama_index.embeddings import SimilarityMode
   service_context = ServiceContext.from_defaults(embed_model="local")
   evaluator = SemanticSimilarityEvaluator(
       service_context=service_context,
       similarity_mode=SimilarityMode.DEFAULT,
       similarity_threshold=0.6,
   )
   response = "The sky is yellow."
   reference = "The sky is blue."
   result = await evaluator.aevaluate(
       response=response,
       reference=reference,
   )
   print("Score: ", result.score)
   print("Passing: ", result.passing)
   Score:  0.9178505509625874
   Passing:  True
We note here that a high score does not imply the answer is always
correct.
Embedding similarity primarily captures the notion of "relevancy".
Since both the response and reference discuss "the sky" and colors,
they are semantically similar.


BatchEvalRunner - Running Multiple Evaluations

The "BatchEvalRunner" class can be used to run a series of evaluations
asynchronously. The async jobs are limited to a defined size of
"num_workers".
Setup
   # attach to the same event-loop
   import nest_asyncio
   nest_asyncio.apply()
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"
   openai.api_key = os.environ["OPENAI_API_KEY"]
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       ServiceContext,
       Response,
   )
   from llama_index.llms import OpenAI
   from llama_index.evaluation import (
       FaithfulnessEvaluator,
       RelevancyEvaluator,
       CorrectnessEvaluator,
   )
   import pandas as pd
   pd.set_option("display.max_colwidth", 0)
Using GPT-4 here for evaluation
   # gpt-4
   gpt4 = OpenAI(temperature=0, model="gpt-4")
   service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)
   faithfulness_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)
   relevancy_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)
   correctness_gpt4 = CorrectnessEvaluator(service_context=service_context_gpt4)
   documents = SimpleDirectoryReader("./test_wiki_data/").load_data()
   # create vector index
   llm = OpenAI(temperature=0.3, model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(llm=llm, chunk_size=512)
   vector_index = VectorStoreIndex.from_documents(
       documents, service_context=service_context
   )
Question Generation
To run evaluations in batch, you can create the runner and then call
the ".aevaluate_queries()" function on a list of queries.
First, we can generate some questions and then run evaluation on them.
   from llama_index.evaluation import DatasetGenerator
   dataset_generator = DatasetGenerator.from_documents(
       documents, service_context=service_context
   )
   questions = dataset_generator.generate_questions_from_nodes(num=25)
Running Batch Evaluation
Now, we can run our batch evaluation!
   from llama_index.evaluation import BatchEvalRunner
   runner = BatchEvalRunner(
       {"faithfulness": faithfulness_gpt4, "relevancy": relevancy_gpt4},
       workers=8,
   )
   eval_results = await runner.aevaluate_queries(
       vector_index.as_query_engine(), queries=questions
   )
   # If we had ground-truth answers, we could also include the correctness evaluator like below.
   # The correctness evaluator depends on additional kwargs, which are passed in as a dictionary.
   # Each question is mapped to a set of kwargs
   #
   # runner = BatchEvalRunner(
   #   {'faithfulness': faithfulness_gpt4, 'relevancy': relevancy_gpt4, 'correctness': correctness_gpt4},
   #   workers=8,
   # )
   #
   # eval_results = await runner.aevaluate_queries(
   #   vector_index.as_query_engine(),
   #   queries=questions,
   #   query_kwargs={'question': {'reference': 'ground-truth answer', ...}}
   # )
Inspecting Outputs
   print(eval_results.keys())
   print(eval_results["faithfulness"][0].dict().keys())
   print(eval_results["faithfulness"][0].passing)
   print(eval_results["faithfulness"][0].response)
   print(eval_results["faithfulness"][0].contexts)
   dict_keys(['faithfulness', 'relevancy'])
   dict_keys(['query', 'contexts', 'response', 'passing', 'feedback', 'score'])
   True
   The population of New York City as of 2020 is 8,804,190.

["== Demographics ==\n\nNew York City is the most populous city in the United States, with 8,804,190 residents incorporating more immigration into the city than outmigration since the 2010 United States census. More than twice as many people live in New York City as compared to Los Angeles, the second-most populous U.S. city; and New York has more than three times the population of Chicago, the third-most populous U.S. city. New York City gained more residents between 2010 and 2020 (629,000) than any other U.S. city, and a greater amount than the total sum of the gains over the same decade of the next four largest U.S. cities, Los Angeles, Chicago, Houston, and Phoenix, Arizona combined. New York City's population is about 44% of New York State's population, and about 39% of the population of the New York metropolitan area. The majority of New York City residents in 2020 (5,141,538, or 58.4%) were living on Long Island, in Brooklyn, or in Queens. The New York City metropolitan statistical area, has the largest foreign-born population of any metropolitan region in the world. The New York region continues to be by far the leading metropolitan gateway for legal immigrants admitted into the United States, substantially exceeding the combined totals of Los Angeles and Miami.\n\n\n=== Population density ===\n\nIn 2020, the city had an estimated population density of 29,302.37 inhabitants per square mile (11,313.71/km2), rendering it the nation's most densely populated of all larger municipalities (those with more than 100,000 residents), with several small cities (of fewer than 100,000) in adjacent Hudson County, New Jersey having greater density, as per the 2010 census. Geographically co-extensive with New York County, the borough of Manhattan's 2017 population density of 72,918 inhabitants per square mile (28,154/km2) makes it the highest of any county in the United States and higher than the density of any individual American city. The next three densest counties in the United States, placing second through fourth, are also New York boroughs: Brooklyn, the Bronx, and Queens respectively.", "New York, often called New York City or NYC, is the most populous city in the United States. With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), New York City is the most densely populated major city in the United States and more than twice as populous as Los Angeles, the nation's second-largest city. New York City is located at the southern tip of New York State. It constitutes the geographical and demographic center of both the Northeast megalopolis and the New York metropolitan area, the largest metropolitan area in the U.S. by both population and urban area. With over 20.1 million people in its metropolitan statistical area and 23.5 million in its combined statistical area as of 2020, New York is one of the world's most populous megacities, and over 58 million people live within 250 mi (400 km) of the city. New York City is a global cultural, financial, entertainment, and media center with a significant influence on commerce, health care and life sciences, research, technology, education, politics, tourism, dining, art

Reporting Total Scores
   def get_eval_results(key, eval_results):
       results = eval_results[key]
       correct = 0
       for result in results:
           if result.passing:
               correct += 1
       score = correct / len(results)
       print(f"{key} Score: {score}")
       return score
   score = get_eval_results("faithfulness", eval_results)
   faithfulness Score: 1.0
   score = get_eval_results("relevancy", eval_results)
   relevancy Score: 0.96


Faithfulness Evaluator

This notebook uses the "FaithfulnessEvaluator" module to measure if
the response from a query engine matches any source nodes.This is
useful for measuring if the response was hallucinated.The data is
extracted from the New York City wikipedia page.
   # attach to the same event-loop
   import nest_asyncio
   nest_asyncio.apply()
   # configuring logger to INFO level
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       TreeIndex,
       VectorStoreIndex,
       SimpleDirectoryReader,
       ServiceContext,
       Response,
   )
   from llama_index.llms import OpenAI
   from llama_index.evaluation import FaithfulnessEvaluator
   import pandas as pd
   pd.set_option("display.max_colwidth", 0)
Using GPT-4 here for evaluation
   # gpt-4
   gpt4 = OpenAI(temperature=0, model="gpt-4")
   service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)
   evaluator_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)
   documents = SimpleDirectoryReader("./test_wiki_data/").load_data()
   # create vector index
   service_context = ServiceContext.from_defaults(chunk_size=512)
   vector_index = VectorStoreIndex.from_documents(
       documents, service_context=service_context
   )
   # define jupyter display function
   def display_eval_df(response: Response, eval_result: str) -> None:
       if response.source_nodes == []:
           print("no response!")
           return
       eval_df = pd.DataFrame(
           {
               "Response": str(response),
               "Source": response.source_nodes[0].node.text[:1000] + "...",
               "Evaluation Result": "Pass" if eval_result.passing else "Fail",
           },
           index=[0],
       )
       eval_df = eval_df.style.set_properties(
           **{
               "inline-size": "600px",
               "overflow-wrap": "break-word",
           },
           subset=["Response", "Source"]
       )
       display(eval_df)
To run evaluations you can call the ".evaluate_response()" function on
the "Response" object return from the query to run the evaluations.
Lets evaluate the outputs of the vector_index.
   query_engine = vector_index.as_query_engine()
   response_vector = query_engine.query("How did New York City get its name?")
   eval_result = evaluator_gpt4.evaluate_response(response=response_vector)
   display_eval_df(response_vector, eval_result)
   <pandas.io.formats.style.Styler at 0x7f991be1bc40>
Benchmark on Generated Question
Now lets generate a few more questions so that we have more to
evaluate with and run a small benchmark.
   from llama_index.evaluation import DatasetGenerator
   question_generator = DatasetGenerator.from_documents(documents)
   eval_questions = question_generator.generate_questions_from_nodes(5)
   eval_questions
   WARNING:llama_index.indices.service_context:chunk_size_limit is deprecated, please specify chunk_size instead
   chunk_size_limit is deprecated, please specify chunk_size instead
   chunk_size_limit is deprecated, please specify chunk_size instead
   ['What is the population of New York City as of 2020?',
    'Which borough of New York City is home to the headquarters of the United Nations?',
    'How many languages are spoken in New York City, making it the most linguistically diverse city in the world?',
    'Who founded the trading post on Manhattan Island that would later become New York City?',
    'What was New York City named after in 1664?']
   import asyncio
   def evaluate_query_engine(query_engine, questions):
       c = [query_engine.aquery(q) for q in questions]

Faithfulness Evaluator

       results = asyncio.run(asyncio.gather(*c))
       print("finished query")
       total_correct = 0
       for r in results:
           # evaluate with gpt 4
           eval_result = 1 if evaluator_gpt4.evaluate_response(response=r).passing else 0
           total_correct += eval_result
       return total_correct, len(results)
   vector_query_engine = vector_index.as_query_engine()
   correct, total = evaluate_query_engine(vector_query_engine, eval_questions[:5])
   print(f"score: {correct}/{total}")
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=35 request_id=b36e17a843c31e827f0b7034e603cf28 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=35 request_id=b36e17a843c31e827f0b7034e603cf28 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=35 request_id=b36e17a843c31e827f0b7034e603cf28 response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=35 request_id=5acb726518065db9312da9f23beef411 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=35 request_id=5acb726518065db9312da9f23beef411 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=35 request_id=5acb726518065db9312da9f23beef411 response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=46 request_id=4af43bfbe4e24fdae0ec33312ee7491e response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=46 request_id=4af43bfbe4e24fdae0ec33312ee7491e response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=46 request_id=4af43bfbe4e24fdae0ec33312ee7491e response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=37 request_id=e30413546fe5f96d3890606767f2ec53 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=37 request_id=e30413546fe5f96d3890606767f2ec53 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=37 request_id=e30413546fe5f96d3890606767f2ec53 response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=33 request_id=01f0a8dada4dae80c97a9a412f03b84f response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=33 request_id=01f0a8dada4dae80c97a9a412f03b84f response_code=200

Faithfulness Evaluator

   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=33 request_id=01f0a8dada4dae80c97a9a412f03b84f response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=282 request_id=ed7b1f8ba68ae32b1d8e24e0d0764e86 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=282 request_id=ed7b1f8ba68ae32b1d8e24e0d0764e86 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=282 request_id=ed7b1f8ba68ae32b1d8e24e0d0764e86 response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=820 request_id=b4532c6d665b6cfd644861ed69819cb9 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=820 request_id=b4532c6d665b6cfd644861ed69819cb9 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=820 request_id=b4532c6d665b6cfd644861ed69819cb9 response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=847 request_id=4d9bbc71a95b7e0bb69a048e251772c8 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=847 request_id=4d9bbc71a95b7e0bb69a048e251772c8 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=847 request_id=4d9bbc71a95b7e0bb69a048e251772c8 response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=952 request_id=d1657940d881929d500b1fddc46b5866 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=952 request_id=d1657940d881929d500b1fddc46b5866 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=952 request_id=d1657940d881929d500b1fddc46b5866 response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=1482 request_id=c4456f75580d227f846d3a044e5eef1b response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=1482 request_id=c4456f75580d227f846d3a044e5eef1b response_code=200

Faithfulness Evaluator

   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=1482 request_id=c4456f75580d227f846d3a044e5eef1b response_code=200
   finished query
   score: 5/5


Retrieval Evaluation

This notebook uses our "RetrieverEvaluator" to evaluate the quality of
any Retriever module defined in LlamaIndex.
We specify a set of different evaluation metrics: this includes hit-
rate and MRR. For any given question, these will compare the quality
of retrieved results from the ground-truth context.
To ease the burden of creating the eval dataset in the first place, we
can rely on synthetic data generation.
Setup
Here we load in data (PG essay), parse into Nodes. We then index this
data using our simple vector index and get a retriever.
   import nest_asyncio
   nest_asyncio.apply()
   from llama_index.evaluation import generate_question_context_pairs
   from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
   from llama_index.node_parser import SimpleNodeParser
   from llama_index.llms import OpenAI
   documents = SimpleDirectoryReader("../../data/paul_graham/").load_data()
   node_parser = SimpleNodeParser.from_defaults(chunk_size=512)
   nodes = node_parser.get_nodes_from_documents(documents)
   # by default, the node ids are set to random uuids. To ensure same id's per run, we manually set them.
   for idx, node in enumerate(nodes):
       node.id_ = f"node_{idx}"
   llm = OpenAI(model="gpt-4")
   service_context = ServiceContext.from_defaults(llm=llm)
   vector_index = VectorStoreIndex(nodes, service_context=service_context)
   retriever = vector_index.as_retriever(similarity_top_k=2)
Try out Retrieval
We'll try out retrieval over a simple dataset.
   retrieved_nodes = retriever.retrieve("What did the author do growing up?")
   from llama_index.response.notebook_utils import display_source_node
   for node in retrieved_nodes:
       display_source_node(node, source_length=1000)
**Node ID:** 749c5544-13ae-4632-b8dd-c6367b718a73**Similarity:**
0.8203777233851344**Text:** What I Worked On
February 2021
Before college the two main things I worked on, outside of school,
were writing and programming. I didn't write essays. I wrote what
beginning writers were supposed to write then, and probably still are:
short stories. My stories were awful. They had hardly any plot, just
characters with strong feelings, which I imagined made them deep.
The first programs I tried writing were on the IBM 1401 that our
school district used for what was then called "data processing." This
was in 9th grade, so I was 13 or 14. The school district's 1401
happened to be in the basement of our junior high school, and my
friend Rich Draves and I got permission to use it. It was like a mini
Bond villain's lair down there, with all these alien-looking machines
— CPU, disk drives, printer, card reader — sitting up on a raised
floor under bright fluorescent lights.
The language we used was an early version of Fortran. You had to type
programs on punch cards, then stack them in ...
**Node ID:** 6e5d20a0-0c93-4465-9496-5e8318640067**Similarity:**
0.8143566621554992**Text:** [10]
Wow, I thought, there's an audience. If I write something and put it
on the web, anyone can read it. That may seem obvious now, but it was
surprising then. In the print era there was a narrow channel to
readers, guarded by fierce monsters known as editors. The only way to
get an audience for anything you wrote was to get it published as a

Retrieval Evaluation

book, or in a newspaper or magazine. Now anyone could publish
anything.
This had been possible in principle since 1993, but not many people
had realized it yet. I had been intimately involved with building the
infrastructure of the web for most of that time, and a writer as well,
and it had taken me 8 years to realize it. Even then it took me
several years to understand the implications. It meant there would be
a whole new generation of essays. [11]
In the print era, the channel for publishing essays had been
vanishingly small. Except for a few officially anointed thinkers who
went to the right parties in New York, the only people allowed t...
Build an Evaluation dataset of (query, context) pairs
Here we build a simple evaluation dataset over the existing text
corpus.
We use our "generate_question_context_pairs" to generate a set of
(question, context) pairs over a given unstructured text corpus. This
uses the LLM to auto-generate questions from each context chunk.
We get back a "EmbeddingQAFinetuneDataset" object. At a high-level
this contains a set of ids mapping to queries and relevant doc chunks,
as well as the corpus itself.
   from llama_index.evaluation import (
       generate_question_context_pairs,
       EmbeddingQAFinetuneDataset,
   )
   qa_dataset = generate_question_context_pairs(nodes, llm=llm, num_questions_per_chunk=2)
   queries = qa_dataset.queries.values()
   print(list(queries)[2])
   In the context, the author mentions his first experience with programming on a TRS-80. Describe the limitations he faced with this early computer and how he used it to write programs, including a word processor.
   # [optional] save
   qa_dataset.save_json("pg_eval_dataset.json")
   # [optional] load
   qa_dataset = EmbeddingQAFinetuneDataset.from_json("pg_eval_dataset.json")
Use "RetrieverEvaluator" for Retrieval Evaluation
We're now ready to run our retrieval evals. We'll run our
"RetrieverEvaluator" over the eval dataset that we generated.
We define two functions: "get_eval_results" and also "display_results"
that run our retriever over the dataset.
   from llama_index.evaluation import RetrieverEvaluator
   retriever_evaluator = RetrieverEvaluator.from_metric_names(
       ["mrr", "hit_rate"], retriever=retriever
   )
   # try it out on a sample query
   sample_id, sample_query = list(qa_dataset.queries.items())[0]
   sample_expected = qa_dataset.relevant_docs[sample_id]
   eval_result = retriever_evaluator.evaluate(sample_query, sample_expected)
   print(eval_result)
   Query: In the context, the author mentions his early experiences with programming on an IBM 1401. Describe the process he used to run a program on this machine and explain why he found it challenging to create meaningful programs on it.
   Metrics: {'mrr': 1.0, 'hit_rate': 1.0}
   # try it out on an entire dataset
   eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)
   import pandas as pd
   def display_results(name, eval_results):
       """Display results from evaluate."""
       metric_dicts = []
       for eval_result in eval_results:
           metric_dict = eval_result.metric_vals_dict
           metric_dicts.append(metric_dict)
       full_df = pd.DataFrame(metric_dicts)
       hit_rate = full_df["hit_rate"].mean()
       mrr = full_df["mrr"].mean()
       metric_df = pd.DataFrame(
           {"retrievers": [name], "hit_rate": [hit_rate], "mrr": [mrr]}
       )
       return metric_df

Retrieval Evaluation

   display_results("top-2 eval", eval_results)
      retrievers  hit_rate       mrr
   0  top-2 eval  0.833333  0.784722


Chat Engine - Simple Mode REPL

Get started in 3 lines of code
Using GPT3 ("text-davinci-003")
   from llama_index.chat_engine import SimpleChatEngine
   chat_engine = SimpleChatEngine.from_defaults()
   chat_engine.chat_repl()
   ===== Entering Chat REPL =====
   Type "exit" to exit.
   Assistant: Hi there! How can I help you?
   Assistant: Why did the chicken cross the playground?
   To get to the other slide!
   Assistant: I'm sorry you didn't find it funny. Is there something else I can help you with?
   Assistant: Why did the scarecrow win the Nobel Prize?
   Because he was outstanding in his field!
   Assistant: Oh, I'm a lumberjack and I'm okay
   I sleep all night and I work all day
   I cut down trees, I eat my lunch
   I go to the lavatory on Wednesday and Saturday
   Everybody knows my song!
Customize LLM
Use ChatGPT ("gpt-3.5-turbo")
   from llama_index.llms import OpenAI
   from llama_index import ServiceContext
   service_context = ServiceContext.from_defaults(
       llm=OpenAI(temperature=0.0, model="gpt-3.5-turbo")
   )
   from llama_index.chat_engine import SimpleChatEngine
   chat_engine = SimpleChatEngine.from_defaults(service_context=service_context)
   chat_engine.chat_repl()
   model='gpt-3.5-turbo' temperature=0.0 max_tokens=None additional_kwargs={} max_retries=10
   ===== Entering Chat REPL =====
   Type "exit" to exit.
   Assistant: Hello! How can I assist you today?
   Assistant: Sure, here's a joke for you:
   Why don't scientists trust atoms?
   Because they make up everything!
   Assistant: I apologize if the joke didn't meet your expectations. Humor can be subjective, and what one person finds funny, another may not. Is there anything else I can assist you with?
   Assistant: Of course! Here's another joke for you:
   Why don't skeletons fight each other?
   They don't have the guts!
   Assistant: Certainly! Here's a little song for you:
   (Verse 1)
   In a world of endless possibilities,
   I'll sing a song to bring you some ease.
   With melodies that dance upon the air,
   I hope this tune will show you I care.
   (Chorus)
   La la la, a melody so sweet,
   La la la, let the music take the lead.
   Close your eyes and let your worries fade,
   As I sing this song, a serenade.
   (Verse 2)
   Through the highs and lows, the ups and downs,
   I'll sing a song to turn your frown around.
   With harmonies that lift your spirits high,
   I hope this melody will make you sigh.
   (Chorus)
   La la la, a melody so sweet,
   La la la, let the music take the lead.
   Close your eyes and let your worries fade,
   As I sing this song, a serenade.
   (Outro)
   So let the music fill your heart and soul,
   As I sing this song, let your worries go.
   May this melody bring you joy and cheer,
   And remind you that I'm always here.
   I hope you enjoyed the song! Let me know if there's anything else I can do for you.
Streaming Support
   from llama_index.llms import OpenAI
   from llama_index import ServiceContext
   service_context = ServiceContext.from_defaults(
       llm=OpenAI(temperature=0.0, model="gpt-3.5-turbo-0613")
   )
   from llama_index.chat_engine import SimpleChatEngine

Chat Engine - Simple Mode REPL

   chat_engine = SimpleChatEngine.from_defaults(service_context=service_context)
   response = chat_engine.stream_chat("Write me a poem about raining cats and dogs.")
   for token in response.response_gen:
       print(token, end="")
   In a world where whimsy takes its flight,
   Where dreams and reality intertwine,
   A tale unfolds, both strange and bright,
   Of raining cats and dogs, so divine.
   From the heavens, a tempest brews,
   Clouds gather, dark and thick,
   And as the wind begins to choose,
   The sky releases a whimsical trick.
   Down they fall, with paws and tails,
   Cats and dogs, in a watery dance,
   Tiny meows and barks prevail,
   As they descend in a wild romance.
   The felines, graceful, land with poise,
   Their fur glistening, sleek and fine,
   With eyes that gleam like emerald joys,
   They prance and purr, in a feline line.
   The canines, playful, splash and bound,
   Their wagging tails a joyful sight,
   With tongues that pant and ears that sound,
   They frolic and bark, with all their might.
   Together they create a symphony,
   A chorus of meows and barks,
   A spectacle for all to see,
   As they dance upon the parks.
   Children giggle, adults stare,
   Amazed by this peculiar sight,
   For in this moment, they're all aware,
   Of the magic raining from the height.
   And as the storm begins to wane,
   The cats and dogs return above,
   Leaving behind a world untamed,
   A memory of a rain so rare and of love.
   So, let us cherish this whimsical tale,
   Of raining cats and dogs, so grand,
   For in the extraordinary, we prevail,
   And find enchantment in the palm of our hand.


Chat Engine - Context Mode

ContextChatEngine is a simple chat mode built on top of a retriever
over your data.
For each chat interaction:
* first retrieve text from the index using the user message
* set the retrieved text as context in the system prompt
* return an answer to the user message
This approach is simple, and works for questions directly related to
the knowledge base and general interactions.
Get started in 5 lines of code
Load data and build index
   import openai
   import os
   os.environ["OPENAI_API_KEY"] = "API_KEY_HERE"
   openai.api_key = os.environ["OPENAI_API_KEY"]
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   data = SimpleDirectoryReader(input_dir="../data/paul_graham/").load_data()
   index = VectorStoreIndex.from_documents(data)
Configure chat engine
Since the context retrieved can take up a large amount of the
available LLM context, let's ensure we configure a smaller limit to
the chat history!
   from llama_index.memory import ChatMemoryBuffer
   memory = ChatMemoryBuffer.from_defaults(token_limit=1500)
   chat_engine = index.as_chat_engine(
       chat_mode="context",
       memory=memory,
       system_prompt="You are a chatbot, able to have normal interactions, as well as talk about an essay discussing Paul Grahams life.",
   )
Chat with your data
   response = chat_engine.chat("Hello!")
   print(response)
   Hello! How can I assist you today?
Ask a follow up question
   response = chat_engine.chat("What did Paul Graham do growing up?")
   print(response)
   Growing up, Paul Graham had a keen interest in writing and programming. He spent a lot of time writing short stories, although he admits that they weren't particularly good. In terms of programming, he started working with computers in 9th grade when he had access to an IBM 1401 computer at his school. He learned an early version of Fortran and experimented with writing programs on punch cards. However, he found it challenging to figure out what to do with the computer since he didn't have much data to work with. It wasn't until microcomputers became available that he truly delved into programming, starting with a kit-built microcomputer called the Heathkit. Eventually, he convinced his father to buy a TRS-80, which allowed him to write simple games, create a word processor, and explore programming further.
   response = chat_engine.chat("Can you tell me more?")
   print(response)
   Certainly! As Paul Graham continued to explore programming, he became fascinated with the possibilities it offered. He enjoyed the process of creating something out of nothing and the logical thinking required in programming. During his high school years, he also developed an interest in painting and considered pursuing it as a career.
   After high school, Paul Graham attended Cornell University, where he studied philosophy. However, he found himself spending more time programming than studying philosophy. He even started a company called Viaweb with some friends, which aimed to create an online store builder. Viaweb eventually became successful and was acquired by Yahoo in 1998.
   After the acquisition, Paul Graham moved to California and became a millionaire. However, he soon realized that he was burnt out from the stress of running Viaweb. He decided to leave Yahoo and pursue his passion for painting. He enrolled in the Accademia di Belle Arti in Florence, Italy, to study painting.
   During his time in Florence, Paul Graham immersed himself in the world of art and painting. He experimented with different techniques and styles, particularly focusing on still life paintings. He found joy in closely observing everyday objects and capturing their details on canvas.
   After a year in Florence, Paul Graham returned to the United States and worked at a software company called Interleaf. Although he was not particularly enthusiastic about the job, it provided him with a steady income and allowed him to save money to pursue his dream of attending the Rhode Island School of Design (RISD) to further his studies in painting.

Chat Engine - Context Mode

   Overall, Paul Graham's journey from programming to painting reflects his curiosity and willingness to explore different passions. He has found success in both fields and continues to share his insights and experiences through his writings and lectures.
Reset conversation state
   chat_engine.reset()
   response = chat_engine.chat("Hello! What do you know?")
   print(response)
   Hi there! I know a lot about Paul Graham's life. He is an entrepreneur, programmer, and investor who is best known for co-founding the venture capital firm Y Combinator. He is also the author of several essays on technology and startups, including the influential essay "Hackers and Painters". He has had a long and successful career in the tech industry, and his experiences have shaped his views on entrepreneurship and technology.
Streaming Support
   from llama_index import (
       ServiceContext,
       VectorStoreIndex,
       SimpleDirectoryReader,
       set_global_service_context,
   )
   from llama_index.llms import OpenAI
   service_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-3.5-turbo", temperature=0)
   )
   set_global_service_context(service_context)
   data = SimpleDirectoryReader(input_dir="../data/paul_graham/").load_data()
   index = VectorStoreIndex.from_documents(data)
   chat_engine = index.as_chat_engine(chat_mode="context")
   response = chat_engine.stream_chat("What did Paul Graham do after YC?")
   for token in response.response_gen:
       print(token, end="")
   After stepping down from his role at Y Combinator (YC), Paul Graham focused on pursuing different interests. Initially, he decided to dedicate his time to painting and see how good he could become with focused practice. He spent most of 2014 painting, but eventually ran out of steam and stopped.
   Following his break from painting, Graham returned to writing essays and also resumed working on Lisp, a programming language. He delved into the core of Lisp, which involves writing an interpreter in the language itself. Graham continued to write essays and work on Lisp in the years following his departure from YC.


Chat Engine - Best Mode

The default chat engine mode is "best", which uses the "openai" mode
if you are using an OpenAI model that supports the latest function
calling API, otherwise uses the "react" mode
Get started in 5 lines of code
Load data and build index
   from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
   from llama_index.llms import OpenAI, Anthropic
   service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-4"))
   data = SimpleDirectoryReader(input_dir="../data/paul_graham/").load_data()
   index = VectorStoreIndex.from_documents(data, service_context=service_context)
Configure chat engine
   chat_engine = index.as_chat_engine(chat_mode="best", verbose=True)
Chat with your data
   response = chat_engine.chat("What are the first programs Paul Graham tried writing?")
   === Calling Function ===
   Calling function: query_engine_tool with args: {
     "input": "What are the first programs Paul Graham tried writing?"
   }
   Got output: The first programs Paul Graham tried writing were on the IBM 1401 that their school district used for "data processing." The language he used was an early version of Fortran.
   ========================
   print(response)
   The first programs Paul Graham tried writing were on the IBM 1401 using an early version of Fortran.


Chat Engine - ReAct Agent Mode

ReAct is an agent based chat mode built on top of a query engine over
your data.
For each chat interaction, the agent enter a ReAct loop:
* first decide whether to use the query engine tool and come up with
  appropriate input
* (optional) use the query engine tool and observe its output
* decide whether to repeat or give final response
This approach is flexible, since it can flexibility choose between
querying the knowledge base or not. However, the performance is also
more dependent on the quality of the LLM. You might need to do more
coercing to make sure it chooses to query the knowledge base at right
times, instead of hallucinating an answer.
Get started in 5 lines of code
Load data and build index
   from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
   from llama_index.llms import OpenAI, Anthropic
   service_context = ServiceContext.from_defaults(llm=OpenAI())
   data = SimpleDirectoryReader(input_dir="../data/paul_graham/").load_data()
   index = VectorStoreIndex.from_documents(data, service_context=service_context)
Configure chat engine
   chat_engine = index.as_chat_engine(chat_mode="react", verbose=True)
Chat with your data
   response = chat_engine.chat(
       "Use the tool to answer what did Paul Graham do in the summer of 1995?"
   )
   Thought: I need to use a tool to help me answer the question.
   Action: query_engine_tool
   Action Input: {'input': 'What did Paul Graham do in the summer of 1995?'}
   Observation:
   In the summer of 1995, Paul Graham worked on building a web application for making web applications. He recruited Dan Giffin, who had worked for Viaweb, and two undergrads who wanted summer jobs, and they got to work trying to build what it's now clear is about twenty companies and several open source projects worth of software. The language for defining applications would of course be a dialect of Lisp.
   Response: In the summer of 1995, Paul Graham worked on building a web application for making web applications. He recruited Dan Giffin, who had worked for Viaweb, and two undergrads who wanted summer jobs, and they got to work trying to build what it's now clear is about twenty companies and several open source projects worth of software. The language for defining applications would of course be a dialect of Lisp.

   print(response)
   In the summer of 1995, Paul Graham worked on building a web application for making web applications. He recruited Dan Giffin, who had worked for Viaweb, and two undergrads who wanted summer jobs, and they got to work trying to build what it's now clear is about twenty companies and several open source projects worth of software. The language for defining applications would of course be a dialect of Lisp.
Customize LLM
Use Anthropic ("claude-2")
   service_context = ServiceContext.from_defaults(llm=Anthropic())
Configure chat engine
   chat_engine = index.as_chat_engine(
       service_context=service_context, chat_mode="react", verbose=True
   )
   response = chat_engine.chat("what did Paul Graham do in the summer of 1995?")
   Thought: I need to use a tool to help me answer the question.
   Action: query_engine_tool
   Action Input: {'input': 'what did Paul Graham do in the summer of 1995?'}
   Observation:  Based on the context, in the summer of 1995 Paul Graham:

Chat Engine - ReAct Agent Mode

   - Painted a second still life using the same objects he had used for a previous still life painting.
   - Looked for an apartment to buy in New York, trying to find a neighborhood similar to Cambridge, MA.
   - Realized there wasn't really a "Cambridge of New York" after visiting the actual Cambridge.
   The passage does not mention what Paul Graham did in the summer of 1995 specifically. It talks about painting a second still life at some point and looking for an apartment in New York at some point, but it does not connect those events to the summer of 1995.
   Response: The passage does not provide enough information to know specifically what Paul Graham did in the summer of 1995. It mentions some activities like painting and looking for an apartment in New York, but does not say these occurred specifically in the summer of 1995.

   print(response)
   The passage does not provide enough information to know specifically what Paul Graham did in the summer of 1995. It mentions some activities like painting and looking for an apartment in New York, but does not say these occurred specifically in the summer of 1995.
   response = chat_engine.chat("What did I ask you before?")
   Response:  You asked me "what did Paul Graham do in the summer of 1995?".

   print(response)
    You asked me "what did Paul Graham do in the summer of 1995?".
Reset chat engine
   chat_engine.reset()
   response = chat_engine.chat("What did I ask you before?")
   Response: I'm afraid I don't have any context about previous questions in our conversation. This seems to be the start of a new conversation between us.

   print(response)
   I'm afraid I don't have any context about previous questions in our conversation. This seems to be the start of a new conversation between us.


Chat Engine - Condense Question Mode

Condense question is a simple chat mode built on top of a query engine
over your data.
For each chat interaction:
* first generate a standalone question from conversation context and
  last message, then
* query the query engine with the condensed question for a response.
This approach is simple, and works for questions directly related to
the knowledge base. Since it *always* queries the knowledge base, it
can have difficulty answering meta questions like "what did I ask you
before?"
Get started in 5 lines of code
Load data and build index
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   data = SimpleDirectoryReader(input_dir="../data/paul_graham/").load_data()
   index = VectorStoreIndex.from_documents(data)
Configure chat engine
   chat_engine = index.as_chat_engine(chat_mode="condense_question", verbose=True)
Chat with your data
   response = chat_engine.chat("What did Paul Graham do after YC?")
   Querying with: What was the next step in Paul Graham's career after his involvement with Y Combinator?
   print(response)
   Paul Graham's next step in his career after his involvement with Y Combinator was to take up painting. He spent most of the rest of 2014 painting and then in March 2015 he started working on Lisp again.
Ask a follow up question
   response = chat_engine.chat("What about after that?")
   Querying with: What did Paul Graham do after he started working on Lisp again in March 2015?
   print(response)
   Paul Graham spent the rest of 2015 writing essays and working on his new dialect of Lisp, which he called Arc. He also looked for an apartment to buy and started planning a second still life painting from the same objects.
   response = chat_engine.chat("Can you tell me more?")
   Querying with: What did Paul Graham do after he started working on Lisp again in March 2015?
   print(response)
   Paul Graham spent the rest of 2015 writing essays and working on his new dialect of Lisp, which he called Arc. He also looked for an apartment to buy and started planning for a second still life painting.
Reset conversation state
   chat_engine.reset()
   response = chat_engine.chat("What about after that?")
   Querying with: What happens after the current situation?
   print(response)
   After the current situation, the narrator resumes painting and experimenting with a new kind of still life. He also resumes his old life in New York, now that he is rich. He is able to take taxis and eat in restaurants, which is exciting for a while. He also starts to make connections with other people who are trying to paint in New York.
Streaming Support
   from llama_index import ServiceContext, VectorStoreIndex, SimpleDirectoryReader
   from llama_index.llms import OpenAI
   service_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-3.5-turbo", temperature=0)
   )
   data = SimpleDirectoryReader(input_dir="../data/paul_graham/").load_data()
   index = VectorStoreIndex.from_documents(data, service_context=service_context)
   chat_engine = index.as_chat_engine(chat_mode="condense_question", verbose=True)
   response = chat_engine.stream_chat("What did Paul Graham do after YC?")
   for token in response.response_gen:
       print(token, end="")
   Querying with: What did Paul Graham do after leaving YC?
   After leaving YC, Paul Graham started painting and focused on improving his skills in that area. He then started writing essays again and began working on Lisp.


Chat Engine with a Personality ✨

Default
   from llama_index.chat_engine import SimpleChatEngine
   chat_engine = SimpleChatEngine.from_defaults()
   response = chat_engine.chat("Say something profound and romantic about fourth of July")
   print(response)
   /Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/deeplake/util/check_latest_version.py:32: UserWarning: A newer version of deeplake (3.6.7) is available. It's recommended that you update to the latest version using `pip install -U deeplake`.
     warnings.warn(
   The Fourth of July is a day to celebrate the beauty of freedom and the power of love.
Shakespeare
   from llama_index.chat_engine import SimpleChatEngine
   from llama_index.prompts.system import SHAKESPEARE_WRITING_ASSISTANT
   chat_engine = SimpleChatEngine.from_defaults(
       system_prompt=SHAKESPEARE_WRITING_ASSISTANT
   )
   response = chat_engine.chat("Say something profound and romantic about fourth of July")
   print(response)
   O Fourth of July, a day of joy and mirth,
   Thou art a day of celebration on this blessed earth.
   A day of fireworks and revelry,
   A day of love and unity.
   Let us all come together and celebrate,
   For this day of freedom we do celebrate.
Marketing
   from llama_index.chat_engine import SimpleChatEngine
   from llama_index.prompts.system import MARKETING_WRITING_ASSISTANT
   chat_engine = SimpleChatEngine.from_defaults(system_prompt=MARKETING_WRITING_ASSISTANT)
   response = chat_engine.chat("Say something profound and romantic about fourth of July")
   print(response)
    Fourth of July is a time to celebrate the freedom and independence of our nation. It's a time to reflect on the beauty of our country and the courage of those who fought for our freedom. It's a time to come together and appreciate the beauty of our nation and the people who make it so special.
IRS Tax
   from llama_index.chat_engine import SimpleChatEngine
   from llama_index.prompts.system import IRS_TAX_CHATBOT
   chat_engine = SimpleChatEngine.from_defaults(system_prompt=IRS_TAX_CHATBOT)
   response = chat_engine.chat("Say something profound and romantic about fourth of July")
   print(response)
    I'm sorry, I can only help with any tax-related questions you may have.


Chat Engine - OpenAI Agent Mode

Get started in 5 lines of code
Load data and build index
   from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
   from llama_index.llms import OpenAI
   # Necessary to use the latest OpenAI models that support function calling API
   service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-3.5-turbo-0613"))
   data = SimpleDirectoryReader(input_dir="../data/paul_graham/").load_data()
   index = VectorStoreIndex.from_documents(data, service_context=service_context)
Configure chat engine
   chat_engine = index.as_chat_engine(chat_mode="openai", verbose=True)
Chat with your data
   response = chat_engine.chat("Hi")
   print(response)
   Hello! How can I assist you today?
   response = chat_engine.chat(
       "Use the tool to answer: Who did Paul Graham hand over YC to?"
   )
   print(response)
   === Calling Function ===
   Calling function: query_engine_tool with args: {
     "input": "Who did Paul Graham hand over YC to?"
   }
   Got output: Paul Graham handed over YC to Sam Altman.
   ========================
   Paul Graham handed over Y Combinator (YC) to Sam Altman.
Force chat engine to query the index
NOTE: this is a feature unique to the "openai" chat mode (which uses
the "OpenAIAgent" under the hood).
   response = chat_engine.chat(
       "What did Paul Graham do growing up?", function_call="query_engine_tool"
   )
   === Calling Function ===
   Calling function: query_engine_tool with args: {
     "input": "What did Paul Graham do growing up?"
   }
   Got output: Growing up, Paul Graham worked on writing and programming. He wrote short stories and tried programming on the IBM 1401 computer in his school's basement. He later got a microcomputer and started programming games and a word processor. He initially planned to study philosophy in college but switched to AI. He also started publishing essays online, which became a significant focus for him.
   ========================
   print(response)
   Growing up, Paul Graham had a passion for writing and programming. He wrote short stories and explored programming on the IBM 1401 computer in his school's basement. He later acquired a microcomputer and began programming games and a word processor. While initially intending to study philosophy in college, he ultimately changed his focus to artificial intelligence (AI). Additionally, he started publishing essays online, which became a significant part of his pursuits.


[Beta] Text-to-SQL with PGVector

This notebook demo shows how to perform text-to-SQL with pgvector.
This allows us to jointly do both semantic search and structured
querying, *all* within SQL!
This hypothetically enables more expressive queries than semantic
search + metadata filters.
**NOTE**: This is a beta feature, interfaces might change. But in the
meantime hope you find it useful!
Setup Data
Load Documents
Load in the Lyft 2021 10k document.
   from llama_hub.file.pdf.base import PDFReader
   reader = PDFReader()
   docs = reader.load_data("../data/10k/lyft_2021.pdf")
   from llama_index.node_parser import SimpleNodeParser
   node_parser = SimpleNodeParser.from_defaults()
   nodes = node_parser.get_nodes_from_documents(docs)
   print(nodes[8].get_content(metadata_mode="all"))
Insert data into Postgres + PGVector
Make sure you have all the necessary dependencies installed!
   !pip install psycopg2-binary pgvector asyncpg "sqlalchemy[asyncio]" greenlet
   from pgvector.sqlalchemy import Vector
   from sqlalchemy import insert, create_engine, String, text, Integer
   from sqlalchemy.orm import declarative_base, mapped_column
Establish Connection
~~~~~~~~~~~~~~~~~~~~
   engine = create_engine("postgresql+psycopg2://localhost/postgres")
   with engine.connect() as conn:
       conn.execute(text("CREATE EXTENSION IF NOT EXISTS vector"))
       conn.commit()
Define Table Schema
~~~~~~~~~~~~~~~~~~~
Define as Python class. Note we store the page_label, embedding, and
text.
   Base = declarative_base()
   class SECTextChunk(Base):
       __tablename__ = "sec_text_chunk"
       id = mapped_column(Integer, primary_key=True)
       page_label = mapped_column(Integer)
       file_name = mapped_column(String)
       text = mapped_column(String)
       embedding = mapped_column(Vector(384))
   Base.metadata.drop_all(engine)
   Base.metadata.create_all(engine)
Generate embedding for each Node with a sentence_transformers model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   # get embeddings for each row
   from llama_index.embeddings import HuggingFaceEmbedding
   embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en")
   for node in nodes:
       text_embedding = embed_model.get_text_embedding(node.get_content())
       node.embedding = text_embedding
Insert into Database
~~~~~~~~~~~~~~~~~~~~
   # insert into database
   for node in nodes:
       row_dict = {
           "text": node.get_content(),
           "embedding": node.embedding,
           **node.metadata,
       }
       stmt = insert(SECTextChunk).values(**row_dict)
       with engine.connect() as connection:
           cursor = connection.execute(stmt)
           connection.commit()
Define PGVectorSQLQueryEngine
Now that we've loaded the data into the database, we're ready to setup
our query engine.
Define Prompt
We create a modified version of our default text-to-SQL prompt to
inject awareness of the pgvector syntax. We also prompt it with some
few-shot examples of how to use the syntax (<-->).
**NOTE**: This is included by default in the "PGVectorSQLQueryEngine",
we included it here mostly for visibility!
   from llama_index.prompts import PromptTemplate
   text_to_sql_tmpl = """\
   Given an input question, first create a syntactically correct {dialect} \
   query to run, then look at the results of the query and return the answer. \
   You can order the results by a relevant column to return the most \
   interesting examples in the database.
   Pay attention to use only the column names that you can see in the schema \
   description. Be careful to not query for columns that do not exist. \
   Pay attention to which column is in which table. Also, qualify column names \
   with the table name when needed.

[Beta] Text-to-SQL with PGVector

   IMPORTANT NOTE: you can use specialized pgvector syntax (`<-->`) to do nearest \
   neighbors/semantic search to a given vector from an embeddings column in the table. \
   The embeddings value for a given row typically represents the semantic meaning of that row. \
   The vector represents an embedding representation \
   of the question, given below. Do NOT fill in the vector values directly, but rather specify a \
   `[query_vector]` placeholder. For instance, some select statement examples below \
   (the name of the embeddings column is `embedding`):
   SELECT * FROM items ORDER BY embedding <-> '[query_vector]' LIMIT 5;
   SELECT * FROM items WHERE id != 1 ORDER BY embedding <-> (SELECT embedding FROM items WHERE id = 1) LIMIT 5;
   SELECT * FROM items WHERE embedding <-> '[query_vector]' < 5;
   You are required to use the following format, \
   each taking one line:
   Question: Question here
   SQLQuery: SQL Query to run
   SQLResult: Result of the SQLQuery
   Answer: Final answer here
   Only use tables listed below.
   {schema}
   Question: {query_str}
   SQLQuery: \
   """
   text_to_sql_prompt = PromptTemplate(text_to_sql_tmpl)
Setup LLM, Embedding Model, and Misc.
Besides LLM and embedding model, note we also add annotations on the
table itself. This better helps the LLM understand the column schema
(e.g. by telling it what the embedding column represents) to better do
either tabular querying or semantic search.
   from llama_index import ServiceContext, SQLDatabase
   from llama_index.llms import OpenAI
   from llama_index.query_engine import PGVectorSQLQueryEngine
   sql_database = SQLDatabase(engine, include_tables=["sec_text_chunk"])
   llm = OpenAI(model="gpt-4")
   service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)
   table_desc = """\
   This table represents text chunks from an SEC filing. Each row contains the following columns:
   id: id of row
   page_label: page number
   file_name: top-level file name
   text: all text chunk is here
   embedding: the embeddings representing the text chunk
   For most queries you should perform semantic search against the `embedding` column values, since \
   that encodes the meaning of the text.
   """
   context_query_kwargs = {"sec_text_chunk": table_desc}
   /Users/jerryliu/Programming/gpt_index/llama_index/utilities/sql_wrapper.py:118: SAWarning: Did not recognize type 'vector' of column 'embedding'
     self._metadata.reflect(
Define Query Engine
   query_engine = PGVectorSQLQueryEngine(
       sql_database=sql_database,
       text_to_sql_prompt=text_to_sql_prompt,
       service_context=service_context,
       context_query_kwargs=context_query_kwargs,
   )
Run Some Queries
Now we're ready to run some queries
   response = query_engine.query(
       "Can you tell me about the risk factors described in page 6?",
   )
   /Users/jerryliu/Programming/gpt_index/llama_index/utilities/sql_wrapper.py:166: SAWarning: Did not recognize type 'vector' of column 'embedding'
     for column in self._inspector.get_columns(table_name):
   print(str(response))
   The text on page 6 discusses the impact of the COVID-19 pandemic on the business. It mentions that the pandemic has affected communities in the United States, Canada, and globally. It has also led to significant disruptions in the business, including a decrease in the number of riders and drivers, reduced hours of operation, and increased costs. The text also discusses the company's transportation network, which offers riders seamless, personalized, and on-demand access to a variety of mobility options.

[Beta] Text-to-SQL with PGVector

   print(response.metadata["sql_query"])
   response = query_engine.query(
       "Tell me more about Lyft's real estate operating leases",
   )
   print(str(response))
   Lyft's lease arrangements include vehicle rental agreements that are accounted for as operating leases. These leases do not meet any specific criteria that would categorize them otherwise. The company's leasehold improvements are amortized on a straight-line basis over the shorter of the term of the lease or the useful life of the assets.
   print(response.metadata["sql_query"][:300])
   SELECT * FROM sec_text_chunk WHERE text LIKE '%Lyft%' AND text LIKE '%real estate%' AND text LIKE '%operating leases%' ORDER BY embedding <-> '[-0.06691089272499084, -0.41431307792663574, 0.2750679850578308, 0.19374045729637146, 0.08942480385303497, -0.16577985882759094, 0.399348646402359, 0.3634052
   # looked at returned result
   print(response.metadata["result"])
   [(157, 93, 'lyft_2021.pdf', "Leases that do not meet any of the above criteria are accounted for as operating leases.Lessor\nThe\n Company's lease arrangements include vehicle re ... (4356 characters truncated) ...  realized. Leasehold improvements are amortized on a straight-line basis over the shorter of the term of the lease, or the useful life of the assets.", '[0.16887704,-0.22762142,0.040292107,0.2951868,0.034039058,-0.092776,0.23275128,0.12367551,0.17209437,-0.08910224,0.30044347,0.1590553,0.21984532,-0.1 ... (4111 characters truncated) ... 0.24707487,0.10685501,0.42726353,-0.16156487,-0.2705381,-0.15468368,0.100748956,-0.19910589,-0.06634029,-0.7986131,-0.14139938,0.55980897,0.31352338]')]
   # structured query
   response = query_engine.query(
       "Tell me about the max page number in this table",
   )
   print(str(response))
   The maximum page number in this table is 238.
   print(response.metadata["sql_query"][:300])
   SELECT MAX(page_label) FROM sec_text_chunk;


Router Query Engine

In this tutorial, we define a custom router query engine that selects
one out of several candidate query engines to execute a query.
Setup
   # NOTE: This is ONLY necessary in jupyter notebook.
   # Details: Jupyter runs an event-loop behind the scenes.
   #          This results in nested event-loops when we start an event-loop to make async queries.
   #          This is normally not allowed, we use nest_asyncio to allow it for convenience.
   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().handlers = []
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SummaryIndex,
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
   )
   Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   NumExpr defaulting to 8 threads.
Load Data
We first show how to convert a Document into a set of Nodes, and
insert into a DocumentStore.
   # load documents
   documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   # initialize service context (set chunk size)
   service_context = ServiceContext.from_defaults(chunk_size=1024)
   nodes = service_context.node_parser.get_nodes_from_documents(documents)
   # initialize storage context (by default it's in-memory)
   storage_context = StorageContext.from_defaults()
   storage_context.docstore.add_documents(nodes)
Define Summary Index and Vector Index over Same Data
   summary_index = SummaryIndex(nodes, storage_context=storage_context)
   vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 17038 tokens
Define Query Engines and Set Metadata
   list_query_engine = summary_index.as_query_engine(
       response_mode="tree_summarize",
       use_async=True,
   )
   vector_query_engine = vector_index.as_query_engine()
   from llama_index.tools.query_engine import QueryEngineTool
   list_tool = QueryEngineTool.from_defaults(
       query_engine=list_query_engine,
       description="Useful for summarization questions related to Paul Graham eassy on What I Worked On.",
   )
   vector_tool = QueryEngineTool.from_defaults(
       query_engine=vector_query_engine,
       description="Useful for retrieving specific context from Paul Graham essay on What I Worked On.",
   )
Define Router Query Engine
There are several selectors available, each with some distinct
attributes.
The LLM selectors use the LLM to output a JSON that is parsed, and the
corresponding indexes are queried.
The Pydantic selectors (currently only supported by "gpt-4-0613" and
"gpt-3.5-turbo-0613" (the default)) use the OpenAI Function Call API
to produce pydantic selection objects, rather than parsing raw JSON.
For each type of selector, there is also the option to select 1 index
to route to, or multiple.
PydanticSingleSelector
Use the OpenAI Function API to generate/parse pydantic objects under
the hood for the router selector.
   from llama_index.query_engine.router_query_engine import RouterQueryEngine
   from llama_index.selectors.llm_selectors import LLMSingleSelector, LLMMultiSelector
   from llama_index.selectors.pydantic_selectors import (
       PydanticMultiSelector,
       PydanticSingleSelector,
   )
   query_engine = RouterQueryEngine(

Router Query Engine

Response(response="\nThis document is a reflection on the author's experiences with computers and writing, from his early days of programming on an IBM 1401 to his more recent work on a web application builder. He recounts his experiences with programming, painting, and starting companies, and how he eventually came to write essays about his life and the choices he made.", source_nodes=[NodeWithScore(node=Node(text='\t\t\n\nWhat I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn\'t write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n\nThe first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district\'s 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain\'s lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.\n\nThe language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.\n\nI was puzzled by the 1401. I couldn\'t figure out what to do with it. And in retrospect there\'s not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn\'t have any data stored on punched cards. The only other option was to do things that didn\'t rely on any input, like calculate approximations of pi, but I didn\'t know enough math to do anything interesting of that type. So I\'m not surprised I can\'t remember any programs I wrote, because they can\'t have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn\'t. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager\'s expression made clear.\n\nWith microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping. [1]\n\nThe first of my friends to get a microcomputer built it himself. It was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.\n\nComputers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started programming. I wrote simple games, a program to predict how high my model rockets

Router Query Engine

Router Query Engine

Response(response="\nAfter RICS, Paul Graham decided to focus on Y Combinator and help get the startups through Demo Day. He also started writing essays again and wrote a few that weren't about startups. In November 2014, he ran out of steam while painting and stopped working on it. He then started working on Lisp again in March 2015.", source_nodes=[NodeWithScore(node=Node(text="of my attention. It had already eaten Arc, and was in the process of eating essays too. Either YC was my life's work or I'd have to leave eventually. And it wasn't, so I would.\n\nIn the summer of 2012 my mother had a stroke, and the cause turned out to be a blood clot caused by colon cancer. The stroke destroyed her balance, and she was put in a nursing home, but she really wanted to get out of it and back to her house, and my sister and I were determined to help her do it. I used to fly up to Oregon to visit her regularly, and I had a lot of time to think on those flights. On one of them I realized I was ready to hand YC over to someone else.\n\nI asked Jessica if she wanted to be president, but she didn't, so we decided we'd try to recruit Sam Altman. We talked to Robert and Trevor and we agreed to make it a complete changing of the guard. Up till that point YC had been controlled by the original LLC we four had started. But we wanted YC to last for a long time, and to do that it couldn't be controlled by the founders. So if Sam said yes, we'd let him reorganize YC. Robert and I would retire, and Jessica and Trevor would become ordinary partners.\n\nWhen we asked Sam if he wanted to be president of YC, initially he said no. He wanted to start a startup to make nuclear reactors. But I kept at it, and in October 2013 he finally agreed. We decided he'd take over starting with the winter 2014 batch. For the rest of 2013 I left running YC more and more to Sam, partly so he could learn the job, and partly because I was focused on my mother, whose cancer had returned.\n\nShe died on January 15, 2014. We knew this was coming, but it was still hard when it did.\n\nI kept working on YC till March, to help get that batch of startups through Demo Day, then I checked out pretty completely. (I still talk to alumni and to new startups working on things I'm interested in, but that only takes a few hours a week.)\n\nWhat should I do next? Rtm's advice hadn't included anything about that. I wanted to do something completely different, so I decided I'd paint. I wanted to see how good I could get if I really focused on it. So the day after I stopped working on YC, I started painting. I was rusty and it took a while to get back into shape, but it was at least completely engaging. [18]\n\nI spent most of the rest of 2014 painting. I'd never been able to work so uninterruptedly before, and I got to be better than I had been. Not good enough, but better. Then in November, right in the middle of a painting, I ran out of steam. Up till that point I'd always been curious to see how the painting I was working on would turn out, but suddenly finishing this one seemed like a chore. So I stopped working on it and cleaned my brushes and haven't painted since. So far anyway.\n\nI realize that

Router Query Engine

       selector=PydanticSingleSelector.from_defaults(),
       query_engine_tools=[
           list_tool,
           vector_tool,
       ],
   )
   query_engine.query("What is the summary of the document?")
   Selecting query engine 0: The first choice is specifically related to summarization questions about Paul Graham's essay on What I Worked On..
   > [get_response] Total LLM token usage: 3411 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total LLM token usage: 3411 tokens
   > [get_response] Total embedding token usage: 0 tokens
query_engine.query("What did Paul Graham do after RICS?")
LLMSingleSelector
Use OpenAI (or any other LLM) to parse generated JSON under the hood
to select a sub-index for routing.
   query_engine = RouterQueryEngine(
       selector=LLMSingleSelector.from_defaults(),
       query_engine_tools=[
           list_tool,
           vector_tool,
       ],
   )
   query_engine.query("What is the summary of the document?")
   Selecting query engine 0: It provides a summary of the document..
   > [get_response] Total LLM token usage: 3411 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total LLM token usage: 3411 tokens
   > [get_response] Total embedding token usage: 0 tokens
   query_engine.query("What did Paul Graham do after RICS?")
   Selecting query engine 1: Useful for retrieving specific context from Paul Graham essay on What I Worked On..
   > [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total embedding token usage: 9 tokens
   > [get_response] Total LLM token usage: 1924 tokens
   > [get_response] Total embedding token usage: 0 tokens
PydanticMultiSelector
In case you are expecting queries to be routed to multiple indexes,
you should use a multi selector. The multi selector sends to query to
multiple sub-indexes, and then aggregates all responses using a
summary index to form a complete answer.
   from llama_index import SimpleKeywordTableIndex
   keyword_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)
   keyword_tool = QueryEngineTool.from_defaults(
       query_engine=vector_query_engine,
       description="Useful for retrieving specific context using keywords from Paul Graham essay on What I Worked On.",
   )
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 0 tokens
   query_engine = RouterQueryEngine(
       selector=PydanticMultiSelector.from_defaults(),
       query_engine_tools=[
           list_tool,
           vector_tool,
           keyword_tool,
       ],
   )
   # This query could use either a keyword or vector query engine, so it will combine responses from both
   query_engine.query(
       "What were noteable events and people from the authors time at Interleaf and YC?"
   )
   Selecting query engine 1: Retrieving specific context from Paul Graham essay on What I Worked On..
   > [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total embedding token usage: 18 tokens
   > [get_response] Total LLM token usage: 1995 tokens
   > [get_response] Total embedding token usage: 0 tokens
   Selecting query engine 2: Retrieving specific context using keywords from Paul Graham essay on What I Worked On..
   > [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total embedding token usage: 0 tokens
   > [get_response] Total LLM token usage: 2055 tokens

Router Query Engine

Response(response="\nNotable events and people from the author's time at Interleaf and YC include: \n\nInterleaf: \n- Founding of Interleaf in 1989\n- Acquisition of Interleaf by Lernout & Hauspie in 1999\n- The author's work on Lisp, which led to the development of the Lisp programming language. \n- The author's work on Arc, which led to the development of the Hacker News website. \n\nYC: \n- Founding of YC in 2005\n- Launch of Hacker News in 2006\n- Recruitment of Sam Altman as President in 2013\n- The author's work with Robert Morris, Trevor Blackwell, and Jessica Livingston to create Y Combinator. \n- The author's work with Sam Altman to reorganize YC and make it a lasting organization. \n- The author's work with YC startups to help them succeed. \n- The author's work on Hacker News, which became a major source of stress. \n- The author's work on internal software for YC, written in Arc. \n- The author's work with Kevin Hale, who offered the author unsolicited advice. \n- The author's mother's stroke and death in 2012 and 2014 respectively\n- Author's retirement from YC in 2014\n- Author's decision to take up painting in 2014\n- Author's return to writing essays and Lisp in 2015", source_nodes=[NodeWithScore(node=Node(text="\nNotable events and people from the author's time at Interleaf and YC include: \n\nInterleaf: \n- Founding of Interleaf in 1989\n- Acquisition of Interleaf by Lernout & Hauspie in 1999\n\nYC: \n- Founding of YC in 2005\n- Launch of Hacker News in 2006\n- Recruitment of Sam Altman as President in 2013\n- Author's mother's stroke and death in 2012 and 2014 respectively\n- Author's retirement from YC in 2014\n- Author's decision to take up painting in 2014\n- Author's return to writing essays and Lisp in 2015", doc_id='cd546791-d1e2-420a-9e9c-fde68d2d51dd', embedding=None, doc_hash='0e61517dfdb144c42c1251f3ed80d58fa2c3859a03f9d7a9ae92d513036690c5', extra_info=None, node_info={'start': 0, 'end': 498, '_node_type': <NodeType.TEXT: '1'>}, relationships={<DocumentRelationship.SOURCE: '1'>: '4183ef8b-b14b-4c73-9754-864d64842c1b'}), score=None), NodeWithScore(node=Node(text="\nNotable events and people from the author's time at Interleaf and YC include: \n\nInterleaf: \n- The author's work on Lisp, which led to the development of the Lisp programming language. \n- The author's work on Arc, which led to the development of the Hacker News website. \n\nYC: \n- The author's work with Robert Morris, Trevor Blackwell, and Jessica Livingston to create Y Combinator. \n- The author's work with Sam Altman to reorganize YC and make it a lasting organization. \n- The author's work with YC startups to help them succeed. \n- The author's work on Hacker News, which became a major source of stress. \n- The author's work on internal software for YC, written in Arc. \n- The author's work with Kevin Hale, who offered the author unsolicited advice.", doc_id='cee04688-dbe7-4749-809e-5a3723e61ac7', embedding=None, doc_hash='246f0f5349eab9d4639f1584170456843b8bd47fcf2862c88437e976309e3a57', extra_info=None, node_info={'start': 0, 'end': 755, '_node_type': <NodeType.TEXT: '1'>}, relationships={<DocumentRelationship.SOURCE: '1'>: '283de7d5-81ed-4dc

Router Query Engine

   > [get_response] Total embedding token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 0 tokens
   > [get_response] Total LLM token usage: 658 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total LLM token usage: 658 tokens
   > [get_response] Total embedding token usage: 0 tokens