Update README.md

README.md

@@ -1,3 +1,182 @@
 ---
 license: mit
+library_name: sentence-transformers
+pipeline_tag: text-classification
 ---
+# Model Card for Model ID
+
+This model borrows from Greg Kamradt’s work here: https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb. The idea is to segment text into semantically coherent chunks. The primary goal of this work is to use sentence transformers embeddings to represent the meaning of sentences and detects shifts in meaning to identify potential breakpoints between chunks.
+
+### Model Description
+
+This model aims to segment a text into semantically coherent chunks. It uses sentence-transformers embeddings to represent the meaning of sentences and detect shifts in the meaning to identify potential breakpoints between chunks. There are two primary changes in function from Greg Kamradt excellent original work: 1) Introduce the use of sentence-transformer embeddings rather than OpenAI to provide an entirely open source implementation of semantic chunking, and 2) add functionality to merge smaller chunks with their most semantically similar neighbors to better normalize chunk size. 
+The goal is to use semantic Understanding to enable a model to consider the meaning of text segments rather than purely relying on punctuation or syntax, and to provide flexiblity so that the breakpoint_percentile_threshold and min_chunk_size can be adjusted to influence the granularity of the chunks.
+
+General Outline: 
+
+Preprocessing
+
+- Loading Text: Reads the text from the specified path.
+- Sentence Tokenization: Splits the text into a list of individual sentences using nltk's sentence tokenizer.
+
+Semantic Embeddings
+
+- Model Loading: Loads a pre-trained Sentence Transformer model ( in this case: 'sentence-transformers/all-mpnet-base-v1').
+- Embedding Generation: Converts each sentence into an embedding to represent its meaning.
+
+Sentence Combination:
+
+- Combines each sentence with its neighbors to form slightly larger units, helping the model understand the context in which changes of topic are likely to occur.
+
+Breakpoint Identification
+
+- Cosine Distance: Calculates cosine distances between embeddings of the combined sentences. These distances represent the degree of semantic dissimilarity.
+- Percentile-Based Threshold: Determines a threshold based on a percentile of the distances (e.g., 95th percentile), where higher values indicate more significant semantic shifts.
+- Locating Breaks: Identifies the indices of distances above the threshold, which mark potential breakpoints between chunks.
+ Chunk Creation:
+
+- Splitting at Breakpoints: Divides the original sentences into chunks based on the identified breakpoints.
+
+Chunk Merging:
+
+- Minimum Chunk Size: Defines a minimum number of sentences to consider a chunk valid.
+- Similarity-Based Merging: Merges smaller chunks with their most semantically similar neighbor based on cosine similarity between chunk embeddings.
+
+Output:
+
+- The model ultimately produces a list of text chunks (chunks), each representing a somewhat self-contained, semantically cohesive segment of the original text.
+
+## Usage
+
+Using this chunker is easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+Then you can implement like this:
+
+```python
+---
+license: mit
+---
+import nltk
+from nltk.tokenize import sent_tokenize
+from sentence_transformers import SentenceTransformer
+from sklearn.metrics.pairwise import cosine_similarity
+import numpy as np
+import matplotlib.pyplot as plt
+
+# Text to be chunked
+with open("/path to text") as f:
+  text = f.read()
+
+# Tokenize the text into sentences
+sentences = sent_tokenize(text)
+
+# Generate embeddings for each sentence using sentence-transformers model of choice
+model = SentenceTransformer('sentence-transformers/all-mpnet-base-v1')
+embeddings = model.encode(sentences)
+
+# Combine the sentences with their neighbors
+def combine_sentences(sentences, buffer_size=1):
+  combined_sentences = []
+  for i in range(len(sentences)):
+    combined_sentence = ' '.join(sentences[max(0, i-buffer_size):min(len(sentences), i+1+buffer_size)])
+    combined_sentences.append(combined_sentence)
+  return combined_sentences
+
+combined_sentences = combine_sentences(sentences)
+combined_embeddings = model.encode(combined_sentences)
+
+# Calculate cosine distances between embeddings
+def calculate_cosine_distances(embeddings):
+  distances = []
+  for i in range(len(embeddings) - 1):
+    similarity = cosine_similarity([embeddings[i]], [embeddings[i + 1]])[0][0]
+    distance = 1 - similarity
+    distances.append(distance)
+  return distances
+
+distances = calculate_cosine_distances(combined_embeddings)
+
+# Identify breakpoints
+# Adjust breakpoint threshhold to change the level of dissimilarity between chunk embeddings (higher for greater dissimilarity)
+
+breakpoint_percentile_threshold = 95
+breakpoint_distance_threshold = np.percentile(distances, breakpoint_percentile_threshold)
+breakpoint_indices = [i for i, distance in enumerate(distances) if distance > breakpoint_distance_threshold]
+
+# Create chunks based on breakpoints
+chunks = []
+start_index = 0
+for breakpoint_index in breakpoint_indices:
+  chunk = ' '.join(sentences[start_index:breakpoint_index + 1])
+  chunks.append(chunk)
+  start_index = breakpoint_index + 1
+chunks.append(' '.join(sentences[start_index:]))
+
+# Set a minimum number of sentences per chunk
+min_chunk_size = 3
+
+# Merge small chunks with their most semantically similar neighbor
+def merge_small_chunks_with_neighbors(chunks, embeddings):
+  merged_chunks = [chunks[0]] # Start with the first chunk
+  merged_embeddings = [embeddings[0]] # And its embedding
+
+  for i in range(1, len(chunks) - 1): # Iterate through chunks, excluding the first and last
+    # If the current chunk is small, consider merging it with a neighbor
+    if len(chunks[i].split('. ')) < min_chunk_size:
+      prev_similarity = cosine_similarity([embeddings[i]], [merged_embeddings[-1]])[0][0]
+      next_similarity = cosine_similarity([embeddings[i]], [embeddings[i + 1]])[0][0]
+
+      # Merge with the most similar neighbor
+      if prev_similarity > next_similarity:
+        merged_chunks[-1] += ' ' + chunks[i]
+        merged_embeddings[-1] = (merged_embeddings[-1] + embeddings[i]) / 2
+      else:
+        chunks[i + 1] = chunks[i] + ' ' + chunks[i + 1]
+        embeddings[i + 1] = (embeddings[i] + embeddings[i + 1]) / 2
+    else:
+      merged_chunks.append(chunks[i])
+      merged_embeddings.append(embeddings[i])
+
+  merged_chunks.append(chunks[-1])
+  merged_embeddings.append(embeddings[-1])
+
+  return merged_chunks, merged_embeddings
+
+# Generate embeddings for each initial chunk and merge most semantically similar neighbors
+chunk_embeddings = model.encode(chunks)
+chunks, chunk_embeddings = merge_small_chunks_with_neighbors(chunks, chunk_embeddings)
+
+print(chunks[0])
+```
+## Evaluation Results
+
+Testing on various buffer sizes and breakpoints using the King James Version of the book of Romans (Available here: https://quod.lib.umich.edu/cgi/k/kjv/kjv-idx?type=DIV1&byte=5015363).
+
+Intra-chunk similarity (how similar the sentences in a given chunk are to each other. Higher = More Semantically Similar): 
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/646f846edcddb2358a256744/JtjTFuh2DhEQCDwkOrrkb.png)
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/646f846edcddb2358a256744/15RQ0-Lu8PvQ1IxxKJsGM.png)
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/646f846edcddb2358a256744/K3XfmdKXyg75n1-77DK0X.png)
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/646f846edcddb2358a256744/ojbecOyGaxDY1PTW1WOY4.png)
+
+Inter-chunk similarity (how similar the respective chunks are to each other. Lower = Less Semantically Similar): 
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/646f846edcddb2358a256744/bI02qstwYwmom5Kfae34N.png)
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/646f846edcddb2358a256744/MhLrbp_AuXJMtbrrPoIiO.png)
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/646f846edcddb2358a256744/Zp-iZF_clPuxHA0CfRiJF.png)
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/646f846edcddb2358a256744/JumSdk0Vxi4zJtiCoA64B.png)
+
+#### Citing and Authors
+
+If you find this model helpful, please enjoy and give all credit to Greg Kamradt for the idea. 
+