Updated application

BuildingAChainlitApp.md

@@ -1,312 +0,0 @@
-# Building a Chainlit App
-
-What if we want to take our Week 1 Day 2 assignment - [Pythonic RAG](https://github.com/AI-Maker-Space/AIE4/tree/main/Week%201/Day%202) - and bring it out of the notebook?
-
-Well - we'll cover exactly that here!
-
-## Anatomy of a Chainlit Application
-
-[Chainlit](https://docs.chainlit.io/get-started/overview) is a Python package similar to Streamlit that lets users write a backend and a front end in a single (or multiple) Python file(s). It is mainly used for prototyping LLM-based Chat Style Applications - though it is used in production in some settings with 1,000,000s of MAUs (Monthly Active Users).
-
-The primary method of customizing and interacting with the Chainlit UI is through a few critical [decorators](https://blog.hubspot.com/website/decorators-in-python).
-
-> NOTE: Simply put, the decorators (in Chainlit) are just ways we can "plug-in" to the functionality in Chainlit. 
-
-We'll be concerning ourselves with three main scopes:
-
-1. On application start - when we start the Chainlit application with a command like `chainlit run app.py`
-2. On chat start - when a chat session starts (a user opens the web browser to the address hosting the application)
-3. On message - when the users sends a message through the input text box in the Chainlit UI
-
-Let's dig into each scope and see what we're doing!
-
-## On Application Start:
-
-The first thing you'll notice is that we have the traditional "wall of imports" this is to ensure we have everything we need to run our application. 
-
-```python
-import os
-from typing import List
-from chainlit.types import AskFileResponse
-from utilities_2.text_utils import CharacterTextSplitter, TextFileLoader
-from utilities_2.openai_utils.prompts import (
-    UserRolePrompt,
-    SystemRolePrompt,
-    AssistantRolePrompt,
-)
-from utilities_2.openai_utils.embedding import EmbeddingModel
-from utilities_2.vectordatabase import VectorDatabase
-from utilities_2.openai_utils.chatmodel import ChatOpenAI
-import chainlit as cl
-```
-
-Next up, we have some prompt templates. As all sessions will use the same prompt templates without modification, and we don't need these templates to be specific per template - we can set them up here - at the application scope. 
-
-```python
-system_template = """\
-Use the following context to answer a users question. If you cannot find the answer in the context, say you don't know the answer."""
-system_role_prompt = SystemRolePrompt(system_template)
-
-user_prompt_template = """\
-Context:
-{context}
-
-Question:
-{question}
-"""
-user_role_prompt = UserRolePrompt(user_prompt_template)
-```
-
-> NOTE: You'll notice that these are the exact same prompt templates we used from the Pythonic RAG Notebook in Week 1 Day 2!
-
-Following that - we can create the Python Class definition for our RAG pipeline - or *chain*, as we'll refer to it in the rest of this walkthrough. 
-
-Let's look at the definition first:
-
-```python
-class RetrievalAugmentedQAPipeline:
-    def __init__(self, llm: ChatOpenAI(), vector_db_retriever: VectorDatabase) -> None:
-        self.llm = llm
-        self.vector_db_retriever = vector_db_retriever
-
-    async def arun_pipeline(self, user_query: str):
-        ### RETRIEVAL
-        context_list = self.vector_db_retriever.search_by_text(user_query, k=4)
-
-        context_prompt = ""
-        for context in context_list:
-            context_prompt += context[0] + "\n"
-
-        ### AUGMENTED
-        formatted_system_prompt = system_role_prompt.create_message()
-
-        formatted_user_prompt = user_role_prompt.create_message(question=user_query, context=context_prompt)
-
-
-        ### GENERATION
-        async def generate_response():
-            async for chunk in self.llm.astream([formatted_system_prompt, formatted_user_prompt]):
-                yield chunk
-
-        return {"response": generate_response(), "context": context_list}
-```
-
-Notice a few things:
-
-1. We have modified this `RetrievalAugmentedQAPipeline` from the initial notebook to support streaming. 
-2. In essence, our pipeline is *chaining* a few events together:
-    1. We take our user query, and chain it into our Vector Database to collect related chunks
-    2. We take those contexts and our user's questions and chain them into the prompt templates
-    3. We take that prompt template and chain it into our LLM call
-    4. We chain the response of the LLM call to the user
-3. We are using a lot of `async` again!
-
-Now, we're going to create a helper function for processing uploaded text files.
-
-First, we'll instantiate a shared `CharacterTextSplitter`.
-
-```python
-text_splitter = CharacterTextSplitter()
-```
-
-Now we can define our helper.
-
-```python
-def process_text_file(file: AskFileResponse):
-    import tempfile
-
-    with tempfile.NamedTemporaryFile(mode="w", delete=False, suffix=".txt") as temp_file:
-        temp_file_path = temp_file.name
-
-    with open(temp_file_path, "wb") as f:
-        f.write(file.content)
-
-    text_loader = TextFileLoader(temp_file_path)
-    documents = text_loader.load_documents()
-    texts = text_splitter.split_text(documents)
-    return texts
-```
-
-Simply put, this downloads the file as a temp file, we load it in with `TextFileLoader` and then split it with our `TextSplitter`, and returns that list of strings!
-
-<div style="border: 2px solid white; padding: 10px; border-radius: 5px; background-color: black; padding: 10px;">
-QUESTION #1:
-
-Why do we want to support streaming? What about streaming is important, or useful?
-
-### ANSWER #1:
-
-Streaming is the continuous transmission of the data from the model to the UI. Instead of waiting and batching up the response into a single
-large message, the response is sent in pieces (streams) as it is created.
-
-The advantages of streaming:
-- quicker initial response - the user sees the first part of the answer sooner
-- it is easier to identify the results are incorrect and terminate the request
-- it is a more natural mode of communication for humans
-- better handling of large data, not requiring complex caching
-- essential for real time processing
-- humans can only read so fast so its an advantage to get some of the data earlier
-
-</div>
-
-## On Chat Start:
-
-The next scope is where "the magic happens". On Chat Start is when a user begins a chat session. This will happen whenever a user opens a new chat window, or refreshes an existing chat window.
-
-You'll see that our code is set-up to immediately show the user a chat box requesting them to upload a file. 
-
-```python
-while files == None:
-        files = await cl.AskFileMessage(
-            content="Please upload a Text File file to begin!",
-            accept=["text/plain"],
-            max_size_mb=2,
-            timeout=180,
-        ).send()
-```
-
-Once we've obtained the text file - we'll use our processing helper function to process our text!
-
-After we have processed our text file - we'll need to create a `VectorDatabase` and populate it with our processed chunks and their related embeddings!
-
-```python
-vector_db = VectorDatabase()
-vector_db = await vector_db.abuild_from_list(texts)
-```
-
-Once we have that piece completed - we can create the chain we'll be using to respond to user queries!
-
-```python
-retrieval_augmented_qa_pipeline = RetrievalAugmentedQAPipeline(
-        vector_db_retriever=vector_db,
-        llm=chat_openai
-    )
-```
-
-Now, we'll save that into our user session!
-
-> NOTE: Chainlit has some great documentation about [User Session](https://docs.chainlit.io/concepts/user-session). 
-
-<div style="border: 2px solid white; padding: 10px; border-radius: 5px; background-color: black; padding: 10px;">
-
-### QUESTION #2: 
-
-Why are we using User Session here? What about Python makes us need to use this? Why not just store everything in a global variable?
-
-### ANSWER #2:
-The application hopefully will be run by many people, at the same time. If the data was stored in a global variable
-this would be accessed by everyone using the application. So everytime someone started a new session, the information 
-would be overwritten, meaning everyone would basically get the same results. Unless only one person used the system 
-at a time.
-
-So the goal is to keep each users session information separate from all the other users. The ChainLit User session
-provides the capability of storing each users data separately.
-</div>
-
-## On Message
-
-First, we load our chain from the user session:
-
-```python
-chain = cl.user_session.get("chain")
-```
-
-Then, we run the chain on the content of the message - and stream it to the front end - that's it!
-
-```python
-msg = cl.Message(content="")
-result = await chain.arun_pipeline(message.content)
-
-async for stream_resp in result["response"]:
-    await msg.stream_token(stream_resp)
-```
-
-## 🎉
-
-With that - you've created a Chainlit application that moves our Pythonic RAG notebook to a Chainlit application!
-
-## 🚧 CHALLENGE MODE 🚧
-
-For an extra challenge - modify the behaviour of your applciation by integrating changes you made to your Pythonic RAG notebook (using new retrieval methods, etc.)
-
-If you're still looking for a challenge, or didn't make any modifications to your Pythonic RAG notebook:
-
-1) Allow users to upload PDFs (this will require you to build a PDF parser as well)
-2) Modify the VectorStore to leverage [Qdrant](https://python-client.qdrant.tech/)
-
-> NOTE: The motivation for these challenges is simple - the beginning of the course is extremely information dense, and people come from all kinds of different technical backgrounds. In order to ensure that all learners are able to engage with the content confidently and comfortably, we want to focus on the basic units of technical competency required. This leads to a situation where some learners, who came in with more robust technical skills, find the introductory material to be too simple - and these open-ended challenges help us do this! 
-
-## Support pdf documents
-
-Code was modified to support pdf documents in the following areas:
-
-1) Change to the request for documents in on_chat_start:
-
-- changed the message to ask for .txt or .pdf file
-- changed the acceptable file formats so that the pdf documents are included in the select pop up
-
-```python
-    while not files:
-        files = await cl.AskFileMessage(
-            content="Please upload a .txt or .pdf file to begin processing!",
-            accept=["text/plain", "application/pdf"],
-            max_size_mb=2,
-            timeout=180,
-        ).send()
-```
-
-2) change process_text_file() function to handle .pdf files
-
-- refactor the code to do all file handling in utilities.text_utils
-- app calls process_file, optionally passing in the text splitter function
-    - default text splitter function is CharacterTextSplitter
-    ```python
-    texts = process_file(file)
-    ```
-- load_file() function does the following
-    - read the uploaded document into a temporary file
-    - identify the file extension
-    - process a .txt file as before resulting in the texts list
-    - if the file is .pdf use the PyMuPDF library to read each page and extract the text and add it to texts list
-    - use the passed in text splitter function to split the documents
-
-```python
-    def load_file(self, file, text_splitter=CharacterTextSplitter()):
-        file_extension = os.path.splitext(file.name)[1].lower()
-        with tempfile.NamedTemporaryFile(mode="wb", delete=False, suffix=file_extension) as temp_file:
-            self.temp_file_path = temp_file.name
-            temp_file.write(file.content)
-
-        if os.path.isfile(self.temp_file_path):
-            if self.temp_file_path.endswith(".txt"):
-                self.load_text_file()
-            elif self.temp_file_path.endswith(".pdf"):
-                self.load_pdf_file()
-            else:
-                raise ValueError(
-                    f"Unsupported file type: {self.temp_file_path}"
-                )
-            return text_splitter.split_text(self.documents)
-        else:
-            raise ValueError(
-                    "Not a file"
-                )
-
-    def load_text_file(self):
-        with open(self.temp_file_path, "r", encoding=self.encoding) as f:
-            self.documents.append(f.read())
-
-    def load_pdf_file(self):
-
-        pdf_document = fitz.open(self.temp_file_path)
-        for page_num in range(len(pdf_document)):
-            page = pdf_document.load_page(page_num)
-            text = page.get_text()
-            self.documents.append(text)
-```
-
-3) Test the handling of .pdf and .txt files
-
-Several different .pdf and .txt files were successfully uploaded and processed by the app
-
-

app.py

@@ -1,14 +1,16 @@
+import chainlit as cl
 import os
+from classes.app_state import AppState
+from classes.model_run_state import ModelRunState
 from dotenv import load_dotenv
-import chainlit as cl
+from langchain.schema.runnable import RunnablePassthrough
 from langchain_openai import ChatOpenAI
-from langchain.prompts import PromptTemplate
-from utilities.rag_utilities import create_vector_store
-from langchain_core.prompts import ChatPromptTemplate
+from langchain_openai.embeddings import OpenAIEmbeddings
+from langchain.embeddings import HuggingFaceEmbeddings
 from operator import itemgetter
-from langchain.schema.output_parser import StrOutputParser
-from langchain.schema.runnable import RunnablePassthrough
-from classes.app_state import AppState
+from utilities.doc_utilities import get_documents
+from utilities.templates import get_qa_prompt
+from utilities.vector_utilities import create_vector_store
 
 document_urls = [
     "https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf",
@@ -21,56 +23,39 @@ load_dotenv()
 # Get the OpenAI API key from environment variables
 openai_api_key = os.getenv("OPENAI_API_KEY")
 
-# Setup our state
-state = AppState()
-state.set_debug(False)
+# Setup our state and read the documents
+app_state = AppState()
+app_state.set_debug(False)
+app_state.set_document_urls(document_urls)
 
-state.set_document_urls(document_urls)
+get_documents(app_state)
 
-state.set_llm_model("gpt-3.5-turbo")
-state.set_embedding_model("text-embedding-3-small")
-state.set_chunk_size(1000)
-state.set_chunk_overlap(100)
+# set up this model run
+chainlit_state = ModelRunState()
+chainlit_state.name = "Chainlit"
 
-# Initialize the OpenAI LLM using LangChain
-llm = ChatOpenAI(model=state.llm_model, openai_api_key=openai_api_key)
-state.set_main_llm(llm)
+chainlit_state.qa_model_name = "gpt-4o-mini" 
+chainlit_state.qa_model = ChatOpenAI(model=chainlit_state.qa_model_name, openai_api_key=openai_api_key)
 
-qdrant_retriever = create_vector_store(state)
+hf_username = "rchrdgwr"
+hf_repo_name = "finetuned-arctic-model-2"
+finetuned_model_name = f"{hf_username}/{hf_repo_name}"
 
-system_template = """
-    You are an expert at explaining technical documents to people.
-    You are provided context below to answer the question.
-    Only use the information provided below.
-    If they do not ask a question, have a conversation with them and ask them if they have any questions
-    If you cannot answer the question with the content below say 'I don't have enough information, sorry'
-    The two documents are 'Blueprint for an AI Bill of Rights' and 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile'
-"""
-human_template = """ 
-===
-question:
-{question}
-
-===
-context:
-{context}
-===
-"""
-chat_prompt = ChatPromptTemplate.from_messages([
-    ("system", system_template),
-    ("human", human_template)
-])
-# create the chain
-openai_chat_model = ChatOpenAI(model="gpt-4o")
+chainlit_state.embedding_model_name = finetuned_model_name
+chainlit_state.embedding_model = HuggingFaceEmbeddings(model_name=chainlit_state.embedding_model_name)
+
+chainlit_state.chunk_size = 1000
+chainlit_state.chunk_overlap = 100
+create_vector_store(app_state, chainlit_state )
 
+chat_prompt = get_qa_prompt()
 
+# create the chain
 
 retrieval_augmented_qa_chain = (
-    {"context": itemgetter("question") | qdrant_retriever, "question": itemgetter("question")}
+    {"context": itemgetter("question") | chainlit_state.retriever, "question": itemgetter("question")}
     | RunnablePassthrough.assign(context=itemgetter("context"))
-
-
-    | {"response": chat_prompt | openai_chat_model, "context": itemgetter("context")}
+    | {"response": chat_prompt | chainlit_state.qa_model, "context": itemgetter("context")}
 )
 
 opening_content = """
@@ -116,19 +101,18 @@ async def main(message):
 
     await cl.Message(content=context_msg).send()
 
-    for doc in context_documents:
-        document_title = doc.metadata.get("source", "Unknown Document") 
-        document_id = doc.metadata.get("document_id", "Unknown ID") 
-        chunk_number = doc.metadata.get("chunk_number", "Unknown Chunk")  
-
-        document_context = doc.page_content.strip() 
-        truncated_context = document_context[:MAX_PREVIEW_LENGTH] + ("..." if len(document_context) > MAX_PREVIEW_LENGTH else "")
-        print("----------------------------------------")
-        print(truncated_context)
-
-        await cl.Message(
-            content=f"**{document_title} ( Chunk: {chunk_number})**",
-            elements=[
-                cl.Text(content=truncated_context, display="inline")  
-            ]
-        ).send()
+    # for doc in context_documents:
+    #     document_title = doc.metadata.get("source", "Unknown Document") 
+    #     chunk_number = doc.metadata.get("chunk_number", "Unknown Chunk")  
+
+    #     document_context = doc.page_content.strip() 
+    #     truncated_context = document_context[:MAX_PREVIEW_LENGTH] + ("..." if len(document_context) > MAX_PREVIEW_LENGTH else "")
+    #     print("----------------------------------------")
+    #     print(truncated_context)
+
+    #     await cl.Message(
+    #         content=f"**{document_title} ( Chunk: {chunk_number})**",
+    #         elements=[
+    #             cl.Text(content=truncated_context, display="inline")  
+    #         ]
+    #     ).send()

classes/app_state.py

@@ -1,86 +1,16 @@
+import pprint
 class AppState:
     def __init__(self):
         self.debug = False
-        self.llm_model = "gpt-3.5-turbo"
-        self.embedding_model = "text-embedding-3-small"
-        self.chunk_size = 1000
-        self.chunk_overlap = 100
         self.document_urls = []
         self.download_folder = "data/"
-        self.loaded_documents = []
-        self.single_text_documents = []
-        self.metadata = []
-        self.titles = []
         self.documents = []
-        self.combined_document_objects = []
-        self.main_llm = None
-        self.retriever = None
 
-        self.system_template = "You are a helpful assistant"
-        #
-        self.user_input = None
-        self.retrieved_documents = []
-        self.chat_history = []
-        self.current_question = None
-    
+    def display(self):
+        pprint.pprint(self.__dict__)
     def set_document_urls(self, document_urls):
         self.document_urls = document_urls
-
-    def set_llm_model(self, llm_model):
-        self.llm_model = llm_model
-
-    def set_embedding_model(self, embedding_model):
-        self.embedding_model = embedding_model
-
-    def set_chunk_size(self, chunk_size):
-        self.chunk_size = chunk_size
-
-    def set_chunk_overlap(self, chunk_overlap):
-        self.chunk_overlap = chunk_overlap
-
-    def set_system_template(self, system_template):
-        self.system_template = system_template
-
-    def add_loaded_document(self, loaded_document):
-        self.loaded_documents.append(loaded_document)
-
-    def add_single_text_documents(self, single_text_document):
-        self.single_text_documents.append(single_text_document)
-    def add_metadata(self, metadata):
-        self.metadata = metadata
-
-    def add_title(self, title):
-        self.titles.append(title)
     def add_document(self, document):
         self.documents.append(document)
-    def add_combined_document_objects(self, combined_document_objects):
-        self.combined_document_objects = combined_document_objects
-    def set_retriever(self, retriever):
-        self.retriever = retriever
-    def set_main_llm(self, main_llm):
-        self.main_llm = main_llm
     def set_debug(self, debug):
-        self.debug = debug
-    #
-    # Method to update the user input
-    def set_user_input(self, input_text):
-        self.user_input = input_text
-
-    # Method to add a retrieved document
-    # def add_document(self, document):
-    #     print("adding document")
-    #     print(self)
-    #     self.retrieved_documents.append(document)
-
-    # Method to update chat history
-    def update_chat_history(self, message):
-        self.chat_history.append(message)
-
-    # Method to get the current state
-    def get_state(self):
-        return {
-            "user_input": self.user_input,
-            "retrieved_documents": self.retrieved_documents,
-            "chat_history": self.chat_history,
-            "current_question": self.current_question
-        }
+        self.debug = debug

classes/model_run_state.py

@@ -0,0 +1,50 @@
+import pprint
+
+from utilities.constants import (
+    CHUNKING_STRATEGY_RECURSIVE,
+    CHUNKING_STRATEGY_TABLE_AWARE,
+    CHUNKING_STRATEGY_SECTION_BASED
+)
+
+class ModelRunState:
+    def __init__(self):
+        self.name = ""
+
+        self.qa_model_name = "gpt-4o"
+        self.qa_model = None
+
+        self.embedding_model_name = "text-embedding-3-small"
+        self.embedding_model = None
+
+        self.chunking_strategy = CHUNKING_STRATEGY_RECURSIVE
+        self.chunk_size = 1000
+        self.chunk_overlap = 100
+
+        self.response_dataset = []
+
+        self.combined_document_objects = []
+        self.retriever = None
+        
+        self.ragas_results = None
+        self.system_template = "You are a helpful assistant"
+    
+    def display(self):
+        pprint.pprint(self.__dict__)
+
+    def parameters(self):
+        print(f"Base model: {self.qa_model_name}")
+        print(f"Embedding model: {self.embedding_model_name}")
+        print(f"Chunk size: {self.chunk_size}")
+        print(f"Chunk overlap: {self.chunk_overlap}")
+
+    def results_summary(self):
+        print(self.ragas_results)
+
+    def results(self):
+        results_df = self.ragas_results.to_pandas()
+        results_df
+
+    @classmethod
+    def compare_ragas_results(cls, model_run_1, model_run_2):
+        if not isinstance(model_run_1, cls) or not isinstance(model_run_2, cls):
+            raise ValueError("Both instances must be of the same class")

classes/ragas_state.py

@@ -0,0 +1,17 @@
+import pprint
+from ragas.testset.evolutions import simple, reasoning, multi_context
+class RagasState:
+    def __init__(self):
+        self.chunk_size = 600
+        self.chunk_overlap = 50
+        self.chunks = []
+        self.generator_llm = "gpt-4"
+        self.critic_llm = "gpt-4o-mini"
+        self.distributions = {
+            simple: 0.5,
+            multi_context: 0.4,
+            reasoning: 0.1
+        }
+        self.num_questions = 3
+        self.testset_df = None
+

old_app.py

@@ -1,145 +0,0 @@
-import os
-from chainlit.types import AskFileResponse
-
-from utilities_2.openai_utils.prompts import (
-    UserRolePrompt,
-    SystemRolePrompt,
-    AssistantRolePrompt,
-)
-from utilities_2.openai_utils.embedding import EmbeddingModel
-from utilities_2.vectordatabase import VectorDatabase
-from utilities_2.openai_utils.chatmodel import ChatOpenAI
-import chainlit as cl
-from utilities.text_utils import FileLoader
-from utilities.pipeline import RetrievalAugmentedQAPipeline
-# from utilities.vector_database import QdrantDatabase
-
-
-def process_file(file, use_rct): 
-    fileLoader = FileLoader()
-    return fileLoader.load_file(file, use_rct)
-
-system_template = """\
-Use the following context to answer a users question. 
-If you cannot find the answer in the context, say you don't know the answer. 
-The context contains the text from a document. Refer to it as the document not the context.
-"""
-system_role_prompt = SystemRolePrompt(system_template)
-
-user_prompt_template = """\
-Context:
-{context}
-
-Question:
-{question}
-"""
-user_role_prompt = UserRolePrompt(user_prompt_template)
-
-@cl.on_chat_start
-async def on_chat_start():
-    # get user inputs
-    res = await cl.AskActionMessage(
-        content="Do you want to use Qdrant?",
-        actions=[
-            cl.Action(name="yes", value="yes", label="✅ Yes"),
-            cl.Action(name="no", value="no", label="❌ No"),
-        ],
-    ).send()
-    use_qdrant = False
-    use_qdrant_type = "Local"
-    if res and res.get("value") == "yes":
-        use_qdrant = True
-        local_res = await cl.AskActionMessage(
-                content="Do you want to use local or cloud?",
-                actions=[
-                    cl.Action(name="Local", value="Local", label="✅ Local"),
-                    cl.Action(name="Cloud", value="Cloud", label="❌ Cloud"),
-                ],
-            ).send()
-        if local_res and local_res.get("value") == "Cloud":
-            use_qdrant_type = "Cloud"
-    use_rct = False
-    res = await cl.AskActionMessage(
-        content="Do you want to use RecursiveCharacterTextSplitter?",
-        actions=[
-            cl.Action(name="yes", value="yes", label="✅ Yes"),
-            cl.Action(name="no", value="no", label="❌ No"),
-        ],
-    ).send()
-    if res and res.get("value") == "yes":
-        use_rct = True
-    
-    files = None
-    # Wait for the user to upload a file
-    while not files:
-        files = await cl.AskFileMessage(
-            content="Please upload a .txt or .pdf file to begin processing!",
-            accept=["text/plain", "application/pdf"],
-            max_size_mb=2,
-            timeout=180,
-        ).send()
-
-    file = files[0]
-
-    msg = cl.Message(
-        content=f"Processing `{file.name}`...", disable_human_feedback=True
-    )
-    await msg.send()
-
-    texts = process_file(file, use_rct)
-
-    msg = cl.Message(
-        content=f"Resulted in {len(texts)} chunks", disable_human_feedback=True
-    )
-    await msg.send()
-
-    # decide if to use the dict vector store of the Qdrant vector store
-    
-    # Create a dict vector store
-    if use_qdrant == False:
-        vector_db = VectorDatabase()
-        vector_db = await vector_db.abuild_from_list(texts)
-    else:
-        embedding_model = EmbeddingModel(embeddings_model_name= "text-embedding-3-small", dimensions=1000)
-        if use_qdrant_type == "Local":
-            from utilities.vector_database import QdrantDatabase
-            vector_db = QdrantDatabase(
-                embedding_model=embedding_model 
-            )
-
-            vector_db = await vector_db.abuild_from_list(texts)
-        
-    msg = cl.Message(
-        content=f"The Vector store has been created", disable_human_feedback=True
-    )
-    await msg.send()
-
-    chat_openai = ChatOpenAI()
-
-    # Create a chain
-    retrieval_augmented_qa_pipeline = RetrievalAugmentedQAPipeline(
-        vector_db_retriever=vector_db,
-        llm=chat_openai,
-        system_role_prompt=system_role_prompt,
-        user_role_prompt=user_role_prompt
-    )
-    
-    # Let the user know that the system is ready
-    msg.content = f"Processing `{file.name}` is complete."
-    await msg.update()
-    msg.content = f"You can now ask questions about `{file.name}`."
-    await msg.update()
-    cl.user_session.set("chain", retrieval_augmented_qa_pipeline)
-
-
-@cl.on_message
-async def main(message):
-    chain = cl.user_session.get("chain")
-
-    msg = cl.Message(content="")
-    result = await chain.arun_pipeline(message.content)
-
-    async for stream_resp in result["response"]:
-        await msg.stream_token(stream_resp)
-
-    await msg.send()

utilities/constants.py

@@ -0,0 +1,4 @@
+CHUNKING_STRATEGY_RECURSIVE = "recursive"
+CHUNKING_STRATEGY_TABLE_AWARE = "table_aware"
+CHUNKING_STRATEGY_SECTION_BASED = "section_based"
+CHUNKING_STRATEGY_SEMANTIC = "semantic_based"

utilities/doc_utilities.py

@@ -0,0 +1,54 @@
+from langchain_community.document_loaders import PyMuPDFLoader
+import fitz
+import os
+import requests
+
+from utilities.debugger import dprint
+import uuid
+
+
+
+def download_document(app_state, url, file_name, download_folder):
+    file_path = os.path.join(download_folder, file_name)
+    if not os.path.exists(download_folder):
+        os.makedirs(download_folder)
+    
+    if not os.path.exists(file_path):
+        print(f"Downloading {file_name} from {url}...")
+        response = requests.get(url)
+        if response.status_code == 200:
+            with open(file_path, 'wb') as f:
+                f.write(response.content)
+        else:
+            dprint(app_state, f"Failed to download document from {url}. Status code: {response.status_code}")
+    else:
+        dprint(app_state, f"{file_name} already exists locally.")
+    return file_path
+
+def get_documents(app_state):
+    for url in app_state.document_urls:
+        dprint(app_state, f"Downloading and loading document from {url}...")
+        file_name = url.split("/")[-1]
+        file_path = download_document(app_state, url, file_name, app_state.download_folder)
+        loader = PyMuPDFLoader(file_path)
+        loaded_document = loader.load()
+        single_text_document = "\n".join([doc.page_content for doc in loaded_document])
+        dprint(app_state, f"Number of pages: {len(loaded_document)}")
+        # lets get titles and metadata
+        pdf = fitz.open(file_path)
+        metadata = pdf.metadata
+        title = metadata.get('title', 'Document 1')
+
+        document = {
+            "url": url,
+            "title": title,
+            "metadata": metadata,
+            "loaded_document": loaded_document,
+            "single_text_document": single_text_document,
+            "document_id": str(uuid.uuid4())
+        }
+        app_state.add_document(document)
+        dprint(app_state, f"Title of Document: {title}")
+        dprint(app_state, f"Full metadata for Document 1: {metadata}")
+        pdf.close()
+    print(f"Total documents: {len(app_state.documents)}")

utilities/get_documents.py

@@ -1,33 +0,0 @@
-import requests
-import os
-from langchain.document_loaders import PyMuPDFLoader
-
-# Define the URLs for the documents
-url_1 = "https://example.com/Blueprint-for-an-AI-Bill-of-Rights.pdf"
-url_2 = "https://example.com/NIST.AI.600-1.pdf"
-
-# Define local file paths for storing the downloaded PDFs
-file_path_1 = "data/Blueprint-for-an-AI-Bill-of-Rights.pdf"
-file_path_2 = "data/NIST.AI.600-1.pdf"
-
-# Function to download a file from a URL
-def download_pdf(url, file_path):
-    # Check if the file already exists to avoid re-downloading
-    if not os.path.exists(file_path):
-        print(f"Downloading {file_path} from {url}...")
-        response = requests.get(url)
-        with open(file_path, 'wb') as f:
-            f.write(response.content)
-    else:
-        print(f"{file_path} already exists, skipping download.")
-        
-# Download the PDFs from the URLs
-download_pdf(url_1, file_path_1)
-download_pdf(url_2, file_path_2)
-
-# Load the PDFs using PyMuPDFLoader
-loader_1 = PyMuPDFLoader(file_path_1)
-documents_1 = loader_1.load()
-
-loader_2 = PyMuPDFLoader(file_path_2)
-documents_2 = loader_2.load()

utilities/pipeline.py

@@ -1,27 +0,0 @@
-from utilities_2.vectordatabase import VectorDatabase
-
-class RetrievalAugmentedQAPipeline:
-    def __init__(self, llm, vector_db_retriever: VectorDatabase, 
-                 system_role_prompt, user_role_prompt
-        ) -> None:
-        self.llm = llm
-        self.vector_db_retriever = vector_db_retriever
-        self.system_role_prompt = system_role_prompt
-        self.user_role_prompt = user_role_prompt
-
-    async def arun_pipeline(self, user_query: str):
-        context_list = self.vector_db_retriever.search_by_text(user_query, k=4)
-
-        context_prompt = ""
-        for context in context_list:
-            context_prompt += context[0] + "\n"
-
-        formatted_system_prompt = self.system_role_prompt.create_message()
-
-        formatted_user_prompt = self.user_role_prompt.create_message(question=user_query, context=context_prompt)
-
-        async def generate_response():
-            async for chunk in self.llm.astream([formatted_system_prompt, formatted_user_prompt]):
-                yield chunk
-
-        return {"response": generate_response(), "context": context_list}

utilities/rag_utilities.py

@@ -1,125 +0,0 @@
-from langchain.text_splitter import RecursiveCharacterTextSplitter
-from langchain.docstore.document import Document
-from langchain_community.document_loaders import PyMuPDFLoader
-from langchain_community.vectorstores import Qdrant
-from langchain_openai.embeddings import OpenAIEmbeddings
-import fitz
-import io
-import os
-import requests
-import tiktoken
-from utilities.debugger import dprint
-import uuid
-
-def tiktoken_len(text):
-    tokens = tiktoken.encoding_for_model("gpt-4o").encode(
-        text,
-    )
-    return len(tokens)
-
-def download_document(state, url, file_name, download_folder):
-    file_path = os.path.join(download_folder, file_name)
-    if not os.path.exists(download_folder):
-        os.makedirs(download_folder)
-    
-    if not os.path.exists(file_path):
-        print(f"Downloading {file_name} from {url}...")
-        response = requests.get(url)
-        if response.status_code == 200:
-            with open(file_path, 'wb') as f:
-                f.write(response.content)
-        else:
-            dprint(state, f"Failed to download document from {url}. Status code: {response.status_code}")
-    else:
-        dprint(state, f"{file_name} already exists locally.")
-    return file_path
-
-def get_documents(state):
-    for url in state.document_urls:
-        dprint(state, f"Downloading and loading document from {url}...")
-        file_name = url.split("/")[-1]
-        file_path = download_document(state, url, file_name, state.download_folder)
-        loader = PyMuPDFLoader(file_path)
-        loaded_document = loader.load()
-        single_text_document = "\n".join([doc.page_content for doc in loaded_document])
-        #state.add_loaded_document(loaded_document)  # Append the loaded documents to the list
-        #state.add_single_text_document(single_text_document)
-        dprint(state, f"Number of pages: {len(loaded_document)}")
-        # lets get titles and metadata
-        pdf = fitz.open(file_path)
-        metadata = pdf.metadata
-        title = metadata.get('title', 'Document 1')
-        #state.add_metadata(metadata)
-        #state.add_title(title)
-        document = {
-            "url": url,
-            "title": title,
-            "metadata": metadata,
-            "single_text_document": single_text_document,
-            "document_id": str(uuid.uuid4())
-        }
-        state.add_document(document)
-        dprint(state, f"Title of Document: {title}")
-        dprint(state, f"Full metadata for Document 1: {metadata}")
-        pdf.close()
-    dprint(state, f"documents: {state.documents}")
-
-def create_chunked_documents(state):
-    get_documents(state)
-
-
-    
-    text_splitter = RecursiveCharacterTextSplitter(
-        chunk_size=state.chunk_size,  
-        chunk_overlap=state.chunk_overlap,
-        length_function = tiktoken_len,
-    )
-    combined_document_objects = []
-    dprint(state, "Chunking documents and creating document objects")
-    for document in state.documents:
-        dprint(state, f"processing documend: {document['title']}")
-        text = document["single_text_document"]
-        dprint(state, text)
-        title = document["title"]
-        document_id = document["document_id"]
-        chunks_document = text_splitter.split_text(text)
-        dprint(state, len(chunks_document))
-
-        for chunk_number, chunk in enumerate(chunks_document, start=1):
-            document_objects = Document(
-                page_content=chunk,
-                metadata={
-                    "source": title,
-                    "document_id": document.get("document_id", "default_id"),
-                    "chunk_number": chunk_number  # Add unique chunk number
-                }
-            )
-            combined_document_objects.append(document_objects)
-    state.add_combined_document_objects(combined_document_objects)
-    
-
-def create_vector_store(state, **kwargs):
-    for key, value in kwargs.items():
-        if hasattr(state, key):
-            setattr(state, key, value)
-        else:
-            print(f"Warning: {key} is not an attribute of the state object")
-
-    # Rest of your create_vector_store logic
-    print(f"Chunk size after update: {state.chunk_size}")
-
-
-
-
-    create_chunked_documents(state)
-    embedding_model = OpenAIEmbeddings(model=state.embedding_model)
-
-    qdrant_vectorstore = Qdrant.from_documents(
-        documents=state.combined_document_objects,
-        embedding=embedding_model,
-        location=":memory:" 
-    )
-    qdrant_retriever = qdrant_vectorstore.as_retriever()
-    state.set_retriever(qdrant_retriever)
-    print("Vector store created")
-    return qdrant_retriever

utilities/templates.py

@@ -0,0 +1,27 @@
+
+from langchain_core.prompts import ChatPromptTemplate
+def get_qa_prompt():
+
+    system_template = """
+        You are an expert at explaining technical documents to people.
+        You are provided context below to answer the question.
+        Only use the information provided below.
+        If they do not ask a question, have a conversation with them and ask them if they have any questions
+        If you cannot answer the question with the content below say 'I don't have enough information, sorry'
+        The two documents are 'Blueprint for an AI Bill of Rights' and 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile'
+    """
+    human_template = """ 
+    ===
+    question:
+    {question}
+
+    ===
+    context:
+    {context}
+    ===
+    """
+    chat_prompt = ChatPromptTemplate.from_messages([
+        ("system", system_template),
+        ("human", human_template)
+    ])
+    return chat_prompt

utilities/text_utils.py

@@ -1,103 +0,0 @@
-import os
-from typing import List
-import fitz # pymupdf
-import tempfile
-from utilities_2.text_utils import CharacterTextSplitter
-from langchain_text_splitters import RecursiveCharacterTextSplitter
-
-# load the file
-# handle .txt and .pdf
-        
-class FileLoader:
-    
-    def __init__(self, encoding: str = "utf-8"):       
-        self.documents = []
-        self.encoding = encoding
-        self.temp_file_path = ""
-
-
-    def load_file(self, file, use_rct):
-        if use_rct:
-            text_splitter=MyRecursiveCharacterTextSplitter()
-        else:
-            text_splitter=CharacterTextSplitter()
-        file_extension = os.path.splitext(file.name)[1].lower()
-
-        with tempfile.NamedTemporaryFile(mode="wb", delete=False, suffix=file_extension) as temp_file:
-            self.temp_file_path = temp_file.name
-            temp_file.write(file.content)
-
-        if os.path.isfile(self.temp_file_path):
-            if self.temp_file_path.endswith(".txt"):
-                self.load_text_file()
-            elif self.temp_file_path.endswith(".pdf"):
-                self.load_pdf_file()
-            else:
-                raise ValueError(
-                    f"Unsupported file type: {self.temp_file_path}"
-                )
-            return text_splitter.split_text(self.documents)
-        else:
-            raise ValueError(
-                    "Not a file"
-                )
-
-    def load_text_file(self):
-        with open(self.temp_file_path, "r", encoding=self.encoding) as f:
-            self.documents.append(f.read())
-
-    def load_pdf_file(self):
-        # pymupdf
-        pdf_document = fitz.open(self.temp_file_path)
-        for page_num in range(len(pdf_document)):
-            page = pdf_document.load_page(page_num)
-            text = page.get_text()
-            self.documents.append(text)
-
-class CharacterTextSplitter:
-    def __init__(
-        self,
-        chunk_size: int = 1000,
-        chunk_overlap: int = 200,
-    ):
-        assert (
-            chunk_size > chunk_overlap
-        ), "Chunk size must be greater than chunk overlap"
-
-        self.chunk_size = chunk_size
-        self.chunk_overlap = chunk_overlap
-
-    def split(self, text: str) -> List[str]:
-        chunks = []
-        for i in range(0, len(text), self.chunk_size - self.chunk_overlap):
-            chunks.append(text[i : i + self.chunk_size])
-        return chunks
-
-    def split_text(self, texts: List[str]) -> List[str]:
-        chunks = []
-        for text in texts:
-            chunks.extend(self.split(text))
-        return chunks
-
-    
-
-class MyRecursiveCharacterTextSplitter:
-    # uses langChain.RecursiveCharacterTextSplitter
-    def __init__(
-        self
-    ):
-        self.RCTS = RecursiveCharacterTextSplitter(
-            chunk_size=1000,
-            chunk_overlap=20,
-            length_function=len,
-            separators=["\n\n", "\n", " ", ""]
-        )
-
-    def split_text(self, texts: List[str]) -> List[str]:
-        all_chunks = []
-        for doc in texts:
-            chunks = self.RCTS.split_text(doc)
-            all_chunks.extend(chunks)
-        return all_chunks
-
-

utilities/vector_database.py

@@ -1,105 +0,0 @@
-import numpy as np
-from collections import defaultdict
-from typing import List, Tuple, Callable
-from utilities_2.openai_utils.embedding import EmbeddingModel
-import hashlib
-from qdrant_client import QdrantClient
-from qdrant_client.http.models import PointStruct
-from qdrant_client.models import VectorParams
-import uuid
-
-def cosine_similarity(vector_a: np.array, vector_b: np.array) -> float:
-    """Computes the cosine similarity between two vectors."""
-    dot_product = np.dot(vector_a, vector_b)
-    norm_a = np.linalg.norm(vector_a)
-    norm_b = np.linalg.norm(vector_b)
-    return dot_product / (norm_a * norm_b)
-
-class QdrantDatabase:
-    def __init__(self, embedding_model=None):
-        self.qdrant_client = QdrantClient(location=":memory:")
-        self.collection_name = "my_collection"
-        self.embedding_model = embedding_model or EmbeddingModel(embeddings_model_name= "text-embedding-3-small", dimensions=1000)
-        vector_params = VectorParams(
-            size=self.embedding_model.dimensions,  # vector size 
-            distance="Cosine"
-          )  # distance metric
-        self.qdrant_client.create_collection(
-            collection_name=self.collection_name,
-            vectors_config={"text": vector_params},
-        )
-        self.vectors = defaultdict(np.array)  # Still keeps a local copy if needed
-
-    def string_to_int_id(self, s: str) -> int:
-        return int(hashlib.sha256(s.encode('utf-8')).hexdigest(), 16) % (10**8)
-    def get_test_vector(self):
-        retrieved_vector = self.qdrant_client.retrieve(
-                collection_name="my_collection",
-                ids=[self.string_to_int_id("test_key")]
-            )
-        return retrieved_vector
-    def insert(self, key: str, vector: np.array) -> None:
-        point_id = str(uuid.uuid4())
-        payload = {"text": key}  
-
-        point = PointStruct(
-            id=point_id,
-            vector={"default": vector.tolist()},
-            payload=payload
-        )
-        print(f"Inserting vector for key: {key}, ID: {point_id}")
-        # Insert the vector into Qdrant with the associated document
-        self.qdrant_client.upsert(
-            collection_name=self.collection_name,
-            points=[point]  # Qdrant expects a list of PointStruct
-        )
-
-
-    def search(
-        self,
-        query_vector: np.array,
-        k: int=5,
-        distance_measure: Callable = cosine_similarity,
-    ) -> List[Tuple[str, float]]:
-        # Perform search in Qdrant
-        if isinstance(query_vector, np.ndarray):
-            query_vector = query_vector.tolist()
-        print(type(query_vector)) 
-        search_results = self.qdrant_client.search(
-            collection_name=self.collection_name,
-            query_vector=query_vector,  # Pass the vector as a list
-            limit=k
-        )
-        return [(result.payload['text'], result.score) for result in search_results]
-
-    def search_by_text(
-        self,
-        query_text: str,
-        k: int,
-        distance_measure: Callable = cosine_similarity,
-        return_as_text: bool = False,
-    ) -> List[Tuple[str, float]]:
-        
-        query_vector = self.embedding_model.get_embedding(query_text)
-        results = self.search(query_vector, k, distance_measure)
-        return [result[0] for result in results] if return_as_text else results
-
-    async def abuild_from_list(self, list_of_text: List[str]) -> "QdrantDatabase":
-        from qdrant_client.http import models
-        embeddings = await self.embedding_model.async_get_embeddings(list_of_text)
-        points = [
-            models.PointStruct(
-            id=str(uuid.uuid4()),
-            vector={"text": embedding}, # Should be a named vector as per vector_config
-            payload={
-                "text": text
-                }
-            )
-            for text, embedding in zip(list_of_text, embeddings)
-        ]
-        self.qdrant_client.upsert(
-            collection_name=self.collection_name,
-            points=points
-        )
-        return self
-    

utilities/vector_utilities.py

@@ -0,0 +1,203 @@
+from utilities.constants import (
+    CHUNKING_STRATEGY_TABLE_AWARE,
+    CHUNKING_STRATEGY_SECTION_BASED,
+    CHUNKING_STRATEGY_SEMANTIC
+)
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain.docstore.document import Document
+from langchain_community.vectorstores import Qdrant
+from langchain_openai.embeddings import OpenAIEmbeddings
+import numpy as np
+import pdfplumber
+import re
+from sentence_transformers import SentenceTransformer
+from sklearn.metrics.pairwise import cosine_similarity
+import tiktoken
+from utilities.debugger import dprint
+
+def create_vector_store(app_state, model_run_state, **kwargs):
+    for key, value in kwargs.items():
+        if hasattr(model_run_state, key):
+            setattr(model_run_state, key, value)
+        else:
+            print(f"Warning: {key} is not an attribute of the state object")
+
+    # Rest of your create_vector_store logic
+    dprint(app_state, f"Chunk size after update: {model_run_state.chunk_size}")
+    create_chunked_documents(app_state, model_run_state)
+
+    qdrant_vectorstore = Qdrant.from_documents(
+        documents=model_run_state.combined_document_objects,
+        embedding=model_run_state.embedding_model,
+        location=":memory:" 
+    )
+    qdrant_retriever = qdrant_vectorstore.as_retriever()
+    model_run_state.retriever = qdrant_retriever
+    print("Vector store created")
+
+def tiktoken_len(text):
+    tokens = tiktoken.encoding_for_model("gpt-4o").encode(
+        text,
+    )
+    return len(tokens)
+
+def create_chunked_documents(app_state, model_run_state):
+    dprint(app_state, model_run_state.chunking_strategy)
+    if model_run_state.chunking_strategy == CHUNKING_STRATEGY_TABLE_AWARE:
+        combined_document_objects = chunk_with_table_aware(app_state, model_run_state)
+    elif model_run_state.chunking_strategy == CHUNKING_STRATEGY_SECTION_BASED:
+        combined_document_objects = chunk_with_section_based(app_state, model_run_state)
+    elif model_run_state.chunking_strategy == CHUNKING_STRATEGY_SEMANTIC:
+        combined_document_objects = chunk_with_semantic_splitter(app_state, model_run_state)
+    else:
+        combined_document_objects = chunk_with_recursive_splitter(app_state, model_run_state)
+    model_run_state.combined_document_objects = combined_document_objects
+    dprint(app_state, "Chunking completed successfully")
+
+
+def chunk_with_recursive_splitter(app_state, model_run_state):    
+    text_splitter = RecursiveCharacterTextSplitter(
+        chunk_size=model_run_state.chunk_size,  
+        chunk_overlap=model_run_state.chunk_overlap,
+        length_function = tiktoken_len,
+    )
+    combined_document_objects = []
+    dprint(app_state, "Chunking documents and creating document objects")
+    for document in app_state.documents:
+        dprint(app_state, f"processing documend: {document['title']}")
+        text = document["single_text_document"]
+        dprint(app_state, text)
+        title = document["title"]
+        # document_id = document["document_id"]
+        chunks_document = text_splitter.split_text(text)
+        dprint(app_state, len(chunks_document))
+
+        for chunk_number, chunk in enumerate(chunks_document, start=1):
+            document_objects = Document(
+                page_content=chunk,
+                metadata={
+                    "source": title,
+                    "document_id": document.get("document_id", "default_id"),
+                    "chunk_number": chunk_number  # Add unique chunk number
+                }
+            )
+            combined_document_objects.append(document_objects)
+    return combined_document_objects
+    
+def chunk_with_table_aware(app_state, model_run_state):
+    combined_document_objects = []
+    dprint(app_state, "Using Table-Aware Chunking for documents.")
+
+    for document in app_state.documents:
+        title = document["title"]
+        text = document["single_text_document"]
+
+        # Check if document is a PDF and contains tables
+        if document.get("is_pdf", False):
+            with pdfplumber.open(document["file_path"]) as pdf:
+                for page in pdf.pages:
+                    tables = page.extract_tables()
+                    for table in tables:
+                        table_content = "\n".join([str(row) for row in table])
+                        document_objects = Document(
+                            page_content=table_content,
+                            metadata={
+                                "source": title,
+                                "document_id": document.get("document_id", "default_id"),
+                                "chunk_number": "table"
+                            }
+                        )
+                        combined_document_objects.append(document_objects)
+        
+        # Chunk the rest of the text
+        text_splitter = RecursiveCharacterTextSplitter(chunk_size=model_run_state.chunk_size, chunk_overlap=model_run_state.chunk_overlap)
+        chunks_document = text_splitter.split_text(text)
+
+        for chunk_number, chunk in enumerate(chunks_document, start=1):
+            document_objects = Document(
+                page_content=chunk,
+                metadata={
+                    "source": title,
+                    "document_id": document.get("document_id", "default_id"),
+                    "chunk_number": chunk_number
+                }
+            )
+            combined_document_objects.append(document_objects)
+
+    return combined_document_objects
+
+
+def chunk_with_section_based(app_state, model_run_state):
+    combined_document_objects = []
+    dprint(app_state, "Using Section-Based Chunking for documents.")
+
+    for document in app_state.documents:
+        text = document["single_text_document"]
+        title = document["title"]
+
+        # Split the text by headings
+        sections = re.split(r"\n[A-Z].+?\n", text)
+
+        # Chunk each section
+        text_splitter = RecursiveCharacterTextSplitter(chunk_size=model_run_state.chunk_size, chunk_overlap=model_run_state.chunk_overlap)
+        for section_number, section in enumerate(sections, start=1):
+            chunks_document = text_splitter.split_text(section)
+            for chunk_number, chunk in enumerate(chunks_document, start=1):
+                document_objects = Document(
+                    page_content=chunk,
+                    metadata={
+                        "source": title,
+                        "document_id": document.get("document_id", "default_id"),
+                        "section_number": section_number,
+                        "chunk_number": chunk_number
+                    }
+                )
+                combined_document_objects.append(document_objects)
+
+    return combined_document_objects
+
+
+def chunk_with_semantic_splitter(app_state, model_run_state):
+    # Load pre-trained model for embeddings
+    model = SentenceTransformer('all-MiniLM-L6-v2')
+
+    combined_document_objects = []
+    dprint(app_state, "Using Semantic-Based Chunking for documents.")
+
+    for document in app_state.documents:
+        text = document["single_text_document"]
+        title = document["title"]
+
+        # Split text into sentences or paragraphs
+        sentences = text.split(". ")  # Simple split by sentence (you can refine this)
+        sentence_embeddings = model.encode(sentences)
+
+        # Group sentences into chunks based on semantic similarity
+        chunks = []
+        current_chunk = []
+        for i in range(len(sentences) - 1):
+            current_chunk.append(sentences[i])
+            # Calculate similarity between consecutive sentences
+            sim = cosine_similarity([sentence_embeddings[i]], [sentence_embeddings[i + 1]])[0][0]
+            if sim < 0.7 or len(current_chunk) >= model_run_state.chunk_size:
+                # If similarity is below threshold or chunk size is reached, start a new chunk
+                chunks.append(" ".join(current_chunk))
+                current_chunk = []
+
+        # Add the final chunk
+        if current_chunk:
+            chunks.append(" ".join(current_chunk))
+
+        # Create document objects for the chunks
+        for chunk_number, chunk in enumerate(chunks, start=1):
+            document_objects = Document(
+                page_content=chunk,
+                metadata={
+                    "source": title,
+                    "document_id": document.get("document_id", "default_id"),
+                    "chunk_number": chunk_number
+                }
+            )
+            combined_document_objects.append(document_objects)
+
+    return combined_document_objects

utilities_2/openai_utils/chatmodel.py

@@ -1,45 +0,0 @@
-from openai import OpenAI, AsyncOpenAI
-from dotenv import load_dotenv
-import os
-
-load_dotenv()
-
-
-class ChatOpenAI:
-    def __init__(self, model_name: str = "gpt-4o-mini"):
-        self.model_name = model_name
-        self.openai_api_key = os.getenv("OPENAI_API_KEY")
-        if self.openai_api_key is None:
-            raise ValueError("OPENAI_API_KEY is not set")
-
-    def run(self, messages, text_only: bool = True, **kwargs):
-        if not isinstance(messages, list):
-            raise ValueError("messages must be a list")
-
-        client = OpenAI()
-        response = client.chat.completions.create(
-            model=self.model_name, messages=messages, **kwargs
-        )
-
-        if text_only:
-            return response.choices[0].message.content
-
-        return response
-    
-    async def astream(self, messages, **kwargs):
-        if not isinstance(messages, list):
-            raise ValueError("messages must be a list")
-        
-        client = AsyncOpenAI()
-
-        stream = await client.chat.completions.create(
-            model=self.model_name,
-            messages=messages,
-            stream=True,
-            **kwargs
-        )
-
-        async for chunk in stream:
-            content = chunk.choices[0].delta.content
-            if content is not None:
-                yield content

utilities_2/openai_utils/embedding.py

@@ -1,60 +0,0 @@
-from dotenv import load_dotenv
-from openai import AsyncOpenAI, OpenAI
-import openai
-from typing import List
-import os
-import asyncio
-
-
-class EmbeddingModel:
-    def __init__(self, embeddings_model_name: str = "text-embedding-3-small", dimensions: int = None):
-        load_dotenv()
-        self.openai_api_key = os.getenv("OPENAI_API_KEY")
-        self.async_client = AsyncOpenAI()
-        self.client = OpenAI()
-        self.dimensions = dimensions
-
-        if self.openai_api_key is None:
-            raise ValueError(
-                "OPENAI_API_KEY environment variable is not set. Please set it to your OpenAI API key."
-            )
-        openai.api_key = self.openai_api_key
-        self.embeddings_model_name = embeddings_model_name
-
-    async def async_get_embeddings(self, list_of_text: List[str]) -> List[List[float]]:
-        embedding_response = await self.async_client.embeddings.create(
-            input=list_of_text, model=self.embeddings_model_name, dimensions=self.dimensions
-        )
-
-        return [embeddings.embedding for embeddings in embedding_response.data]
-
-    async def async_get_embedding(self, text: str) -> List[float]:
-        embedding = await self.async_client.embeddings.create(
-            input=text, model=self.embeddings_model_name, dimensions=self.dimensions
-        )
-
-        return embedding.data[0].embedding
-
-    def get_embeddings(self, list_of_text: List[str]) -> List[List[float]]:
-        embedding_response = self.client.embeddings.create(
-            input=list_of_text, model=self.embeddings_model_name, dimensions=self.dimensions
-        )
-
-        return [embeddings.embedding for embeddings in embedding_response.data]
-
-    def get_embedding(self, text: str) -> List[float]:
-        embedding = self.client.embeddings.create(
-            input=text, model=self.embeddings_model_name, dimensions=self.dimensions
-        )
-
-        return embedding.data[0].embedding
-
-
-if __name__ == "__main__":
-    embedding_model = EmbeddingModel()
-    print(asyncio.run(embedding_model.async_get_embedding("Hello, world!")))
-    print(
-        asyncio.run(
-            embedding_model.async_get_embeddings(["Hello, world!", "Goodbye, world!"])
-        )
-    )

utilities_2/openai_utils/prompts.py

@@ -1,78 +0,0 @@
-import re
-
-
-class BasePrompt:
-    def __init__(self, prompt):
-        """
-        Initializes the BasePrompt object with a prompt template.
-
-        :param prompt: A string that can contain placeholders within curly braces
-        """
-        self.prompt = prompt
-        self._pattern = re.compile(r"\{([^}]+)\}")
-
-    def format_prompt(self, **kwargs):
-        """
-        Formats the prompt string using the keyword arguments provided.
-
-        :param kwargs: The values to substitute into the prompt string
-        :return: The formatted prompt string
-        """
-        matches = self._pattern.findall(self.prompt)
-        return self.prompt.format(**{match: kwargs.get(match, "") for match in matches})
-
-    def get_input_variables(self):
-        """
-        Gets the list of input variable names from the prompt string.
-
-        :return: List of input variable names
-        """
-        return self._pattern.findall(self.prompt)
-
-
-class RolePrompt(BasePrompt):
-    def __init__(self, prompt, role: str):
-        """
-        Initializes the RolePrompt object with a prompt template and a role.
-
-        :param prompt: A string that can contain placeholders within curly braces
-        :param role: The role for the message ('system', 'user', or 'assistant')
-        """
-        super().__init__(prompt)
-        self.role = role
-
-    def create_message(self, format=True, **kwargs):
-        """
-        Creates a message dictionary with a role and a formatted message.
-
-        :param kwargs: The values to substitute into the prompt string
-        :return: Dictionary containing the role and the formatted message
-        """
-        if format:
-            return {"role": self.role, "content": self.format_prompt(**kwargs)}
-        
-        return {"role": self.role, "content": self.prompt}
-
-
-class SystemRolePrompt(RolePrompt):
-    def __init__(self, prompt: str):
-        super().__init__(prompt, "system")
-
-
-class UserRolePrompt(RolePrompt):
-    def __init__(self, prompt: str):
-        super().__init__(prompt, "user")
-
-
-class AssistantRolePrompt(RolePrompt):
-    def __init__(self, prompt: str):
-        super().__init__(prompt, "assistant")
-
-
-if __name__ == "__main__":
-    prompt = BasePrompt("Hello {name}, you are {age} years old")
-    print(prompt.format_prompt(name="John", age=30))
-
-    prompt = SystemRolePrompt("Hello {name}, you are {age} years old")
-    print(prompt.create_message(name="John", age=30))
-    print(prompt.get_input_variables())

utilities_2/text_utils.py

@@ -1,75 +0,0 @@
-import os
-from typing import List
-
-class TextFileLoader:
-    def __init__(self, path: str, encoding: str = "utf-8"):
-        self.documents = []
-        self.path = path
-        self.encoding = encoding
-
-    def load(self):
-        if os.path.isdir(self.path):
-            self.load_directory()
-        elif os.path.isfile(self.path) and self.path.endswith(".txt"):
-            self.load_file()
-        else:
-            raise ValueError(
-                "Provided path is neither a valid directory nor a .txt file."
-            )
-
-    def load_file(self):
-        with open(self.path, "r", encoding=self.encoding) as f:
-            self.documents.append(f.read())
-
-    def load_directory(self):
-        for root, _, files in os.walk(self.path):
-            for file in files:
-                if file.endswith(".txt"):
-                    with open(
-                        os.path.join(root, file), "r", encoding=self.encoding
-                    ) as f:
-                        self.documents.append(f.read())
-
-    def load_documents(self):
-        self.load()
-        return self.documents
-
-
-class CharacterTextSplitter:
-    def __init__(
-        self,
-        chunk_size: int = 1000,
-        chunk_overlap: int = 200,
-    ):
-        assert (
-            chunk_size > chunk_overlap
-        ), "Chunk size must be greater than chunk overlap"
-
-        self.chunk_size = chunk_size
-        self.chunk_overlap = chunk_overlap
-
-    def split(self, text: str) -> List[str]:
-        chunks = []
-        for i in range(0, len(text), self.chunk_size - self.chunk_overlap):
-            chunks.append(text[i : i + self.chunk_size])
-        return chunks
-
-    def split_text(self, texts: List[str]) -> List[str]:
-        chunks = []
-        for text in texts:
-            chunks.extend(self.split(text))
-        return chunks
-
-if __name__ == "__main__":
-    loader = TextFileLoader("data/KingLear.txt")
-    loader.load()
-    splitter = CharacterTextSplitter()
-    chunks = splitter.split_text(loader.documents)
-    print(len(chunks))
-    print(chunks[0])
-    print("--------")
-    print(chunks[1])
-    print("--------")
-    print(chunks[-2])
-    print("--------")
-    print(chunks[-1])

utilities_2/vectordatabase.py

@@ -1,82 +0,0 @@
-import numpy as np
-from collections import defaultdict
-from typing import List, Tuple, Callable
-from utilities_2.openai_utils.embedding import EmbeddingModel
-import asyncio
-
-
-def cosine_similarity(vector_a: np.array, vector_b: np.array) -> float:
-    """Computes the cosine similarity between two vectors."""
-    dot_product = np.dot(vector_a, vector_b)
-    norm_a = np.linalg.norm(vector_a)
-    norm_b = np.linalg.norm(vector_b)
-    return dot_product / (norm_a * norm_b)
-
-
-class VectorDatabase:
-    def __init__(self, embedding_model: EmbeddingModel = None):
-        self.vectors = defaultdict(np.array)
-        self.embedding_model = embedding_model or EmbeddingModel()
-
-    def insert(self, key: str, vector: np.array) -> None:
-        self.vectors[key] = vector
-
-    def search(
-        self,
-        query_vector: np.array,
-        k: int,
-        distance_measure: Callable = cosine_similarity,
-    ) -> List[Tuple[str, float]]:
-        scores = [
-            (key, distance_measure(query_vector, vector))
-            for key, vector in self.vectors.items()
-        ]
-        return sorted(scores, key=lambda x: x[1], reverse=True)[:k]
-
-    def search_by_text(
-        self,
-        query_text: str,
-        k: int,
-        distance_measure: Callable = cosine_similarity,
-        return_as_text: bool = False,
-    ) -> List[Tuple[str, float]]:
-        query_vector = self.embedding_model.get_embedding(query_text)
-        results = self.search(query_vector, k, distance_measure)
-        return [result[0] for result in results] if return_as_text else results
-
-    def retrieve_from_key(self, key: str) -> np.array:
-        return self.vectors.get(key, None)
-
-    async def abuild_from_list(self, list_of_text: List[str]) -> "VectorDatabase":
-        embeddings = await self.embedding_model.async_get_embeddings(list_of_text)
-        for text, embedding in zip(list_of_text, embeddings):
-            self.insert(text, np.array(embedding))
-        return self
-
-
-
-if __name__ == "__main__":
-    list_of_text = [
-        "I like to eat broccoli and bananas.",
-        "I ate a banana and spinach smoothie for breakfast.",
-        "Chinchillas and kittens are cute.",
-        "My sister adopted a kitten yesterday.",
-        "Look at this cute hamster munching on a piece of broccoli.",
-    ]
-
-    vector_db = VectorDatabase()
-    vector_db = asyncio.run(vector_db.abuild_from_list(list_of_text))
-    k = 2
-
-    searched_vector = vector_db.search_by_text("I think fruit is awesome!", k=k)
-    print(f"Closest {k} vector(s):", searched_vector)
-
-    retrieved_vector = vector_db.retrieve_from_key(
-        "I like to eat broccoli and bananas."
-    )
-    print("Retrieved vector:", retrieved_vector)
-
-    relevant_texts = vector_db.search_by_text(
-        "I think fruit is awesome!", k=k, return_as_text=True
-    )
-    print(f"Closest {k} text(s):", relevant_texts)