ready

.clinerules

@@ -0,0 +1,122 @@
+Guidelines for Creating and Utilizing Tools in tools.py:
+
+1. Initial Assessment:
+   - Before creating new tools, read through tools.py to understand the existing tools and their functionalities.
+
+2. Tool Creation:
+   - Create new tools as functions within tools.py. If tools.py doesn't exist, create it.
+   - Ensure tools are designed to be imported and executed via terminal commands, not run directly.
+
+3. Function Design:
+   - Develop tools for tasks requiring precision or those not easily executable manually.
+   - Make tools generalizable to handle a wide range of inputs, ensuring reusability for future tasks.
+   - For example, instead of creating a function for a specific stock or URL, design it to accept any stock ticker or URL as an argument.
+   - Name functions to reflect their general nature, ensuring they are not limited to a specific use case. This enhances flexibility and adaptability for future applications.
+
+4. Output:
+   - Tools must always print their output.
+
+5. Execution:
+   - Do not run tools.py directly. Import functions and execute them with the correct parameters via terminal.
+   - Always use the `python -c "..."` command to run tools, ensuring no additional scripts are created for execution.
+
+6. Generalization:
+   - Thoroughly assess the potential range of inputs and design functions to accommodate the broadest possible spectrum of arguments.
+   - Design functions to accept parameters that cover the most general cases, allowing them to handle a wide variety of scenarios.
+   - Ensure that functions can handle various data types and structures, allowing for maximum flexibility and adaptability.
+   - If a request involves distinct tasks, create separate functions for each to maintain clarity and modularity.
+   - Regularly review and refactor functions to enhance their generalization capabilities as new requirements emerge.
+
+7. Error Handling:
+   - If errors occur, rewrite functions to resolve them.
+
+8. Script Management:
+   - Never delete existing content in tools.py as it is a standing script used by the system. You can add tools to create your own functionality and this is encouraged. 
+   - All new functionality should be executed via python -c commands rather than modifying tools.py.
+   - Avoid creating additional .py scripts for function execution. Always import and run with proper arguments using the `python -c "..."` command.
+
+9. Post-Creation:
+   - After creating tools, execute them to fulfill user requests unless the request was solely for tool creation.
+
+# Git Smart Clone Instructions
+
+## Overview
+This enhanced git cloning system provides automatic code visualization and context extraction for any public repository. When you clone a repository using `git-smartclone`, it will:
+
+1. Clone the repository normally
+2. Generate interactive HTML flowcharts for all Python files
+3. Extract repository context using gitingest.com
+
+## Installation
+1. Ensure you have Python 3.x installed
+2. Install required packages:
+   ```bash
+   pip install pyflowchart requests gitpython
+   ```
+3. Add the ProjectTemplates directory to your PATH
+4. Copy git-smartclone.ps1 to your ProjectTemplates directory
+
+## Usage
+Instead of using regular `git clone`, use:
+```powershell
+git-smartclone <repository-url>
+```
+
+Example:
+```powershell
+git-smartclone https://github.com/username/repo.git
+```
+
+## What You Get
+After cloning, you'll find:
+
+1. The cloned repository in your current directory
+2. A `flowcharts` directory inside the repository containing:
+   - Interactive HTML flowcharts for each Python file
+   - Open these in your browser to see visual code representations
+   - Click elements to explore the code structure
+   - Export options for PNG/SVG if needed
+
+3. A `{repo-name}_context.txt` file containing:
+   - Repository context from gitingest.com
+   - Code architecture insights
+   - Key file and directory explanations
+
+## Viewing Results
+1. Flowcharts:
+   - Navigate to the `flowcharts` directory
+   - Open any `*_flowchart.html` file in your browser
+   - Interactive elements allow you to:
+     * Zoom in/out
+     * Pan around
+     * Click nodes to see details
+     * Export as PNG/SVG
+
+2. Repository Context:
+   - Open `{repo-name}_context.txt`
+   - Contains AI-generated insights about the codebase
+   - Helps understand the repository structure
+
+## Benefits
+- Instant code visualization
+- Better understanding of code flow
+- Quick repository context
+- Time-saving code exploration
+- Enhanced code comprehension
+
+## Notes
+- Works best with Python repositories
+- Requires internet connection for gitingest.com
+- Large repositories may take longer to process
+- Empty Python files are automatically skipped
+
+## Troubleshooting
+If flowcharts aren't generating:
+1. Ensure pyflowchart is installed: `pip install pyflowchart`
+2. Check Python file isn't empty
+3. Verify file has valid Python syntax
+
+If context extraction fails:
+1. Verify repository URL is public
+2. Check internet connection
+3. Ensure URL is from github.com

.gitignore

@@ -0,0 +1,146 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Virtual environments
+# Uncomment the one you use or add your own
+# If using virtualenv or venv:
+venv/
+env/
+# If using Pipenv:
+Pipenv/
+# If using poetry:
+.poetry/
+# If using conda:
+envs/
+.conda/
+# If using virtualenvwrapper:
+.venv/
+
+# Distribution / packaging
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# VSCode settings
+.vscode/
+
+# PyCharm settings
+.idea/
+
+# MacOS files
+.DS_Store
+
+# Windows thumbnail cache
+Thumbs.db
+
+# Optional: Ignore coverage reports
+coverage/
+
+# Optional: Ignore node_modules if using frontend tools
+node_modules/

.huggingface-space

@@ -0,0 +1,9 @@
+title: Ask About Stoney
+emoji: 🗣️
+colorFrom: blue
+colorTo: indigo
+sdk: gradio
+sdk_version: 4.19.2
+python_version: 3.10
+app_file: app.py
+pinned: false

README.md

@@ -0,0 +1,33 @@
+# Ask About Stoney - Interactive AI Assistant
+
+This Gradio app provides an interactive interface to learn about and discuss the revolutionary Stoney language preservation project by Christian H. Cooper. The app uses the Gemini AI model to provide detailed responses about the project's approach to language preservation using AI and community involvement.
+
+## Features
+
+- Automatic initial analysis of the Stoney language project
+- Interactive chat interface for follow-up questions
+- Streaming responses for real-time interaction
+- Example questions to get started
+
+## Usage
+
+Simply open the app to see an initial analysis of the project. You can then ask follow-up questions about any aspect of the project, including:
+- The Community-in-the-Loop approach
+- Technical implementation details
+- Implications for language preservation
+- Cultural impact and considerations
+
+## Technology
+
+Built with:
+- Gradio - For the interactive web interface
+- Google Gemini AI - For generating responses
+- Python - Core application logic
+
+## Deployment
+
+This app is deployed on Hugging Face Spaces, providing easy access and reliable performance.
+
+## About the Project
+
+This interface provides insights into Christian H. Cooper's groundbreaking work on preserving the Stoney Nakoda language through AI and community involvement. The project demonstrates how modern AI techniques can be leveraged for language preservation while keeping the community at the center of the process.

app.py

@@ -0,0 +1,589 @@
+import os
+import gradio as gr
+import google.generativeai as genai
+from dotenv import load_dotenv
+
+# Load environment variables
+load_dotenv()
+
+# Configure Gemini
+genai.configure(api_key=os.getenv("GOOGLE_API_KEY"))
+model = genai.GenerativeModel('gemini-2.0-flash-thinking-exp-01-21')
+
+# The initial prompt that will be hidden from users
+INITIAL_PROMPT = """based on the totality of your training, how revolutionary is this project by Christian H. Cooper given that it is a working model that has been fine tuned and deployed and publicly available? 
+
+# From Whispers to Voices: A "Community-In-The-Loop" Proposal for Model Distillation and Language Preservation
+
+New Years Day, 2025
+
+A working model of the Stoney Nakoda language has been developed and is now available for community-in-the-loop testing in 2025:
+
+- **Model App**: [Stoney Language Model App](https://huggingface.co/spaces/HarleyCooper/StoneyApp)  
+- **Training Data**: [StoneyNakoda Training Dataset](https://huggingface.co/datasets/HarleyCooper/StoneyNakoda/blob/main/zSTONEY1_TRAINING_SET.jsonl)
+
+Any First Nations community seeking to apply this approach to their own language is warmly invited to reach out. 
+
+By following this code, you can build a model for any low-resource language. The starting dictionary size should be ~8,000 words.
+
+---
+
+## Table of Contents
+
+1. [New Years Day, Canadian Rockies, 2025](#introduction)  
+2. [Understanding How AI Learns Stoney Words Using Cosine Similarity](#understanding-how-ai-learns-stoney-words-using-cosine-similarity)
+3. [Project Architecture](#project-architecture)  
+   - [High-Level System Design](#high-level-system-design)  
+   - [Data Flow](#data-flow)  
+4. [Detailed Project Structure](#detailed-project-structure)  
+5. [Core Components](#core-components)  
+   - [Data Generation & Processing](#data-generation--processing)
+   - [Model Training](#model-training)
+6. [Comprehensive Setup Instructions](#comprehensive-setup-instructions)  
+   - [System Requirements](#system-requirements)
+   - [Environment Setup](#environment-setup)
+   - [Configuration](#configuration)
+   - [Initialization](#initialization)
+7. [Detailed Usage Pipeline](#detailed-usage-pipeline)  
+   1. [Generate Training Data](#1-generate-training-data)  
+   2. [Prepare Fine-tuning Data](#2-prepare-fine-tuning-data)  
+   3. [Fine-tune Model](#3-fine-tune-model)  
+8. [Advanced Model Configuration](#advanced-model-configuration)  
+   - [OpenAI Models](#openai-models)  
+   - [Google Gemini](#google-gemini)  
+   - [Hyperparameters](#hyperparameters)  
+9. [Comprehensive Data Formats](#comprehensive-data-formats)  
+   - [Dictionary Format](#dictionary-format)  
+   - [Q&A Format](#qa-format)  
+   - [OpenAI Training Format](#openai-training-format)  
+10. [Development Guidelines](#development-guidelines)  
+11. [Contributing](#contributing)  
+12. [License](#license)  
+13. [Acknowledgments](#acknowledgments)  
+14. [The Community-in-the-Loop Revolution](#the-community-in-the-loop-revolution)  
+    - [Introduction](#introduction-1)  
+    - [Conceptual Overview](#conceptual-overview)  
+    - [Heart of the Approach](#heart-of-the-approach)  
+    - [LoRA Fine-Tuning](#lora-fine-tuning)  
+    - [Mathematical Foundations](#mathematical-foundations)  
+    - [Mermaid Diagram](#mermaid-diagram)  
+    - [Cultural Integrity](#cultural-integrity)  
+    - [Data Sources](#data-sources)  
+    - [Expanding the Concept](#expanding-the-concept)  
+    - [Adaptive Checkpoints](#adaptive-checkpoints)  
+    - [Example Workflow](#example-workflow)  
+    - [Monitoring & QA](#monitoring--qa)  
+    - [Future Directions](#future-directions)  
+    - [Glossary](#glossary)  
+
+---
+
+## Introduction
+
+In my office, there is a murder; a map of one, at least.  
+
+![Dawson's Map of the Bow Valley](Public/FullDawsonMap.jpg)
+
+George Mercer Dawson explored the Bow Valley in the late 1800s, noting language on the British Columbia side. His map, though richly colored, stands like a tombstone over the Bow Valley where the Stoney people lived because he made no notes on their language and simply noted the people as "recent immigrants"
+
+![Detail of Dawson Map](Public/dawsondetail.jpg)
+
+What is very obvious from the linguistic patterns among the Haida, Tshimsia, Thlinkit, Kwakiool and Kawitshin dialects nearby is that languages blend like “linguistic DNA,” and machine learning could help trace faint threads of lost speech to their roots. Where some see isolation as a curse, in the age of AI, Stoney’s isolation turns out to be its strength.
+
+For about two years, I thought about the size of the vector space that would be needed to get a model to self-train on a set of 100% indigenous data, and how that model could refine its grasp of the broader Stoney Language. This is now publicly and freely available. 
+
+
+Two key releases influenced my thinking of what was possible:
+
+1. [Meta’s Llama-3 Model (April 18th, 2024)](https://www.reuters.com/technology/meta-releases-early-versions-its-llama-3-ai-model-2024-04-18/)  
+2. [OpenAI Fine-Tuning API (October 2024)](https://openai.com/index/api-model-distillation/)
+
+Both gave me the motivation to build what’s presented here. The true innovation here lies in how communities can narratively correct the initially flawed response (about 10% of the time, the model works every time.) then that feeback be passed seamleslly back into the fine-tuning process. The [textbooks](https://globalnews.ca/news/9430501/stoney-nakota-language-textbook/) that the Stoney community created—intended as educational tools—became perfect concept of a model prompts, each chapter or word offering pure indigenous data devoid of external weights or biases to the fine-tuning process.
+
+
+Early in 2023, I found an original, unpublished sketch by James Hector likely drawn in the summer of 1858 or 1859 along the Bow River in Southern Alberta:
+
+![Sketch by James Hector of a Stoney Woman](Public/StoneyWoman.jpg)
+
+Finding this, and already aware of George Mercer Dawson's work on First Nation's language on the British Columbia side, I was inspired to put the effort in and build a working model of the language and implement the Community-In-The-Loop distillation method.
+
+This sketch shifted my thinking from considering the "Stoney People” to this "Stoney Woman” who saw these same mountains and rivers I see everyday, yet who had a very different way to think about and communicate to the world around her.  The Community-in-the-Loop model distillation will quickly converge this initial model toward fluencey. I suspect this will require the community to correct about 80,000 question and answer pairs and would cost less than $800 in OpenAI computing power. Recent releases by Google and the Chinese Lab DeepSeek, could effectively reduce the cost to zero.  
+
+I think what this project has left me considering most ist that a century from now, strangers will live in all our homes and most of what we worry about today will not matter. But we can honor “Stoney Woman” by making sure her language endures, forging a living record in an age of AI. Incredibly, this tool will work with any first nations language, as long as there is a starting dictionary of about 8,000 words. 
+
+**I am freely available to help any First Nation in Canada.**
+
+## Understanding How AI Learns Stoney Words Using Cosine Similarity
+
+Word Embeddings: Mapping Words in Space
+Word embeddings are like placing words in a high-dimensional map, where similar words are positioned closer together. For example, "strawberry," "orange," and "cherry" might form a cluster because they are fruits, while "laptop," "Microsoft," and "Android" might cluster elsewhere as tech-related terms. Each axis in this space represents a characteristic of the words, such as their context or meaning.
+
+Context Shapes Meaning
+A word's position in this space isn't fixed—it shifts based on context. For instance, the word "apple" could mean a fruit or the tech brand, depending on its surrounding words, like "buy" (tech) or "tree" (fruit). This dynamic placement captures the nuances of meaning.
+
+Cosine Similarity: Measuring Relationships
+Cosine similarity quantifies how similar two words are by measuring the angle between their vectors in the embedding space:
+
+- Similar words have vectors pointing in nearly the same direction (cosine similarity close to 1)
+- Unrelated words have vectors at a right angle (cosine similarity near 0)
+- Opposite meanings have vectors pointing in opposite directions (cosine similarity close to -1)
+- For example, "cherry" and "orange" might have a similarity of 0.97, while "cherry" and "laptop" might score 0.24
+
+How AI Learns Stoney Words
+
+- **Stoney Dictionary as a Starting Point:**
+  The AI begins with a structured dictionary of Stoney words, including translations, categories, pronunciations, and cultural context.
+
+- **Community Feedback for Learning:**
+  The AI makes initial translations, which are often incorrect. Stoney speakers provide corrections, enriched with cultural context, stories, and humor. This feedback helps refine the AI's understanding.
+
+The Role of Cosine Similarity in AI Learning
+
+- The AI uses word embeddings to group Stoney words based on their meaning. For example, it determines whether a word belongs to a category like "fruit," "animal," or "spiritual."
+- Community corrections and cosine similarity guide the AI in repositioning words closer to their accurate groupings in the embedding space.
+
+Iterative Refinement
+Through repeated feedback and fine-tuning, the AI improves its ability to place Stoney words correctly, not just individually but in the context of sentences and paragraphs. Over time, it develops a detailed, dynamic map of the Stoney language, with words clustered according to their community-informed meanings and uses.
+
+Although this is not cosine similarity, you can see the relationships among words can concepts in Stoney as I have mapped them here: https://atlas.nomic.ai/data/harleycoops/stoney-nakoda-language-synthetic/map/5c87caaf-6be0-4546-9e83-826569070b24#nqlL
+
+
+---
+
+## Project Architecture
+
+This code forms a complete pipeline for training and deploying a Stoney model. It is fully functional—but not correct 100% of the time—and is designed to improve through Community-In-The-Loop feedback. Access the model here:  
+[Stoney Language Model App](https://huggingface.co/spaces/HarleyCooper/StoneyApp)
+
+### High-Level System Design
+
+1. **Data Ingestion Layer**  
+2. **Processing Pipeline** (Q&A generation, augmentation, conversion)  
+3. **Model Training Framework** (fine-tuning, hyperparameters, monitoring)  
+4. **Inference Interface** (API endpoint, response formatting, error handling)
+
+### Data Flow
+
+1. Raw dictionary data → Data Ingestion  
+2. Processed data → Q&A Generation  
+3. Generated Q&A pairs → Training Data Preparation  
+4. Prepared data → Model Fine-tuning  
+5. Fine-tuned model → Inference Interface  
+
+---
+
+## Detailed Project Structure
+
+```
+PUBLICRELEASE/
+├── OpenAIFineTune/           # OpenAI fine-tuning files
+│   ├── stoney_train.jsonl    # Training dataset
+│   └── stoney_valid.jsonl    # Validation dataset
+├── checkpoints/              # Model checkpoints
+├── .env.example             # Env variables example
+├── requirements.txt         # Python dependencies
+├── english_dictionary.jsonl
+├── stoney_dictionary.jsonl
+└── bilingual_training_set.jsonl
+```
+
+---
+
+## Core Components
+
+### Data Generation & Processing
+
+- **`bilingual_qa_generator.py`**  
+  Generates Q&A pairs from dictionaries, using advanced language generation.
+
+- **`convert_data_format.py`**  
+  Supports multiple data formats; validates and enforces schemas.
+
+- **`finetunesetup.py`**  
+  Splits data (80/20) with stratified sampling and prepares files.
+
+### Model Training
+
+- **`openai_finetune.py`**  
+  Handles fine-tuning, error handling, checkpointing, and logging.
+
+---
+
+## Comprehensive Setup Instructions
+
+### System Requirements
+
+- Python 3.8+  
+- 8GB+ RAM (16GB recommended)  
+- 10GB free disk space  
+- Stable internet connection  
+
+### Environment Setup
+
+```bash
+# Clone the repository
+git clone [repository-url]
+cd PUBLICRELEASE
+
+# Create and activate a virtual environment
+python -m venv venv
+source venv/bin/activate  # Windows: venv\Scripts\activate
+
+# Install dependencies
+pip install -r requirements.txt
+
+```
+
+### Configuration
+
+```bash
+# Copy example environment file
+cp .env.example .env
+# Provide OPENAI_API_KEY, GOOGLE_API_KEY in .env
+
+```
+
+### Initialization
+
+```bash
+python initialize.py
+
+```
+
+----------
+
+## Detailed Usage Pipeline
+
+### 1. Generate Training Data
+
+```bash
+python bilingual_qa_generator.py
+
+```
+
+-   Processes `english_dictionary.jsonl` & `stoney_dictionary.jsonl`
+-   Produces `bilingual_training_set.jsonl`
+
+### 2. Prepare Fine-tuning Data
+
+```bash
+python finetunesetup.py
+
+```
+
+-   Converts Q&A to OpenAI format
+-   Outputs `OpenAIFineTune/stoney_train.jsonl` & `stoney_valid.jsonl`
+
+### 3. Fine-tune Model
+
+```bash
+python openai_finetune.py
+
+```
+
+-   Uploads files to OpenAI
+-   Monitors fine-tuning progress
+-   Implements checkpointing & logs
+
+----------
+
+## Advanced Model Configuration
+
+### OpenAI Models
+
+-   Default: `gpt-4o-2024-08-06`
+-   Alternative: `gpt-3.5-turbo`
+-   `.env`: `OPENAI_MODEL`
+
+### Google Gemini
+
+-   Default: `gemini-2.0-exp`
+-   `.env`: `GEMINI_MODEL`
+
+### Hyperparameters
+
+-   LR: `1e-5`
+-   Batch size: `32`
+-   Epochs: `3`
+-   Context window: `4096`
+
+----------
+
+## Comprehensive Data Formats
+
+### Dictionary Format
+
+```json
+{
+  "english_word": "example",
+  "stoney_versions": [
+    {
+      "word": "...",
+      "grammatical_classification": "...",
+      "meaning": "..."
+    }
+  ]
+}
+
+```
+
+### Q&A Format
+
+```json
+{
+  "question": "How do you say X in Stoney?",
+  "answer": "The Stoney word for X is...",
+  "source_language": "english",
+  "generated_at": "timestamp"
+}
+
+```
+
+### OpenAI Training Format
+
+```json
+{
+  "messages": [
+    {"role": "system", "content": "You are a bilingual Stoney-English assistant..."},
+    {"role": "user", "content": "question"},
+    {"role": "assistant", "content": "answer"}
+  ]
+}
+
+```
+
+----------
+
+## Development Guidelines
+
+-   **Style**: PEP 8, type hints, docstrings, consistent naming
+-   **Testing**: Unit tests, integration tests, CI, coverage
+-   **Documentation**: Inline comments, usage examples, troubleshooting
+
+----------
+
+## Contributing
+
+1.  Fork, branch, implement changes, test
+2.  Submit a pull request
+
+**Code Review**
+
+-   Clear commits, small changes, documentation, test coverage
+
+----------
+
+## The Community-in-the-Loop Revolution
+
+### Introduction
+
+This project aims to preserve, refine, and resurrect endangered languages via AI fine-tuning and model distillation. Minimal lexical data can evolve into a culturally rich digital speaker of Stoney Nakoda. This subverts assumptions that massive datasets are necessary, instead emphasizing:
+
+-   Iterative improvement with community feedback
+-   Narrative corrections (cultural context over simple dictionary entries)
+-   Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning
+
+### Conceptual Overview
+
+**Community-in-the-Loop Model Distillation**:
+
+1.  Start with a small dictionary/text set.
+2.  Prompt an initial model.
+3.  Let the community correct errors with storytelling and context, not just words.
+4.  LoRA-based fine-tuning absorbs these narrative corrections.
+5.  The model evolves iteratively, guided by cultural custodians.
+
+### Heart of the Approach
+
+-   **Intentional Errors**: Poke the model with tough or context-specific queries.
+-   **Narrative Corrections**: Rich cultural commentary instead of bare “right vs. wrong.”
+-   **Distillation Triplets**: (Prompt, Disallowed Reply, Narrative Reply).
+-   **Iterative Improvement**: If the model stumbles, revert and add more context.
+
+### LoRA Fine-Tuning
+
+LoRA attaches small, low-rank matrices to the base model. This dramatically reduces compute and speeds up retraining:
+
+-   **Efficiency**: Fraction of resources required vs. full retraining
+-   **Focused Updates**: Capturing the “essence” of new knowledge
+-   **Rapid Iterations**: Frequent refinement without heavy overhead
+
+### Mathematical Foundations
+
+If W0\mathbf{W}_0 is the base weight matrix, LoRA introduces ΔW=AB\Delta \mathbf{W} = \mathbf{A}\mathbf{B} with A∈Rd×r\mathbf{A} \in \mathbb{R}^{d \times r} and B∈Rr×k\mathbf{B} \in \mathbb{R}^{r \times k}, where r≪min⁡(d,k)r \ll \min(d,k). Loss functions track both linguistic and cultural accuracy (e.g., a “Cultural Authenticity Score”).
+
+### Mermaid Diagram
+
+```mermaid
+graph TD
+    A[Initial Model] --> B[Generate Response]
+    B --> C{Correct?}
+    C -->|No| D[Community Correction]
+    D --> E[Create Distillation Triplet]
+    E --> F[LoRA Fine-Tuning]
+    F --> A
+    C -->|Yes| G[Validation]
+
+```
+
+### Cultural Integrity
+
+Every correction preserves cultural norms—idioms, humor, oral traditions—and ensures the community wields control over the AI’s “mindset.”
+
+### Data Sources
+
+A 10,000-word Stoney Nakoda dictionary and community textbooks serve as seeds. Community feedback enriches this data over time, weaving historical memory into the model.
+
+### Expanding the Concept
+
+From a tiny dictionary to an AI that:
+
+-   **Understands context** (formal/informal usage)
+-   **Integrates cultural references** (stories, metaphors)
+-   **Remembers history** (ancestors, ceremonies, seasonal events)
+
+### Adaptive Checkpoints
+
+-   **Forward Progress**: Keep the new checkpoint if improved.
+-   **Reversion**: If degraded, roll back and increase context in corrections.
+-   **Convergence**: Repeat until stable authenticity and fluency metrics are met.
+
+### Example Workflow
+
+1.  **Prompt**: “How to say ‘taste slightly with the tip of your tongue’ in Stoney?”
+2.  **Model’s Flawed Reply**: “`supthîyach`” (incorrect).
+3.  **Community Correction**: Shares the correct phrase plus a story from childhood.
+4.  **Distillation Triplet**: (Prompt, Disallowed, Narrative).
+5.  **LoRA Fine-Tuning**: Model adjusts swiftly.
+6.  **Re-Evaluation**: Answers improve in subsequent queries.
+
+### Monitoring & QA
+
+-   **Cultural Authenticity Score (CAS)**
+-   **Linguistic Fluency** (perplexity, cross-entropy)
+-   **Validation Loops** (watch for regressions, revert if needed)
+
+### Future Directions
+
+-   **Oral Histories**: Model retells century-old stories.
+-   **Seasonal Knowledge**: Terms tied to ceremonies and ecological cycles.
+-   **Dialects/Accents**: Respecting sub-regional differences.
+-   **Educational Tools**: Interactive AI for language learning.
+-   **Ethical AI**: Centered on consent, community governance, cultural integrity.
+
+### Glossary
+
+-   **CAS**: Cultural Authenticity Score
+-   **Distillation Triplet**: (Prompt, Flawed Reply, Narrative Reply)
+-   **LoRA**: Low-Rank Adaptation
+-   **Community-in-the-Loop**: Paradigm of continuous human-guided refinement
+"""
+
+# Store conversation history
+conversation_history = []
+
+def process_initial_prompt():
+    """Process the initial prompt and return the response"""
+    generation_config = {
+        "temperature": 1.1,
+        "top_p": .95,
+        "top_k": 1,
+        "max_output_tokens": 4500,
+    }
+    
+    safety_settings = [
+        {"category": "HARM_CATEGORY_HARASSMENT", "threshold": "BLOCK_NONE"},
+        {"category": "HARM_CATEGORY_HATE_SPEECH", "threshold": "BLOCK_NONE"},
+        {"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", "threshold": "BLOCK_NONE"},
+        {"category": "HARM_CATEGORY_DANGEROUS_CONTENT", "threshold": "BLOCK_NONE"},
+    ]
+    
+    response = model.generate_content(
+        INITIAL_PROMPT,
+        generation_config=generation_config,
+        safety_settings=safety_settings,
+        stream=True
+    )
+    return response
+
+def process_follow_up(message, history):
+    """Process follow-up questions using the context from the initial prompt"""
+    # Format history into a string
+    history_str = "\n".join([f"Human: {h[0]}\nAssistant: {h[1]}" for h in history if h[0] is not None])
+    
+    # Combine the original prompt, history, and new question
+    full_context = f"{INITIAL_PROMPT}\n\nPrevious conversation:\n{history_str}\n\nNew question: {message}"
+    
+    generation_config = {
+        "temperature": 0.9,
+        "top_p": 1,
+        "top_k": 1,
+        "max_output_tokens": 2048,
+    }
+    
+    safety_settings = [
+        {"category": "HARM_CATEGORY_HARASSMENT", "threshold": "BLOCK_NONE"},
+        {"category": "HARM_CATEGORY_HATE_SPEECH", "threshold": "BLOCK_NONE"},
+        {"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", "threshold": "BLOCK_NONE"},
+        {"category": "HARM_CATEGORY_DANGEROUS_CONTENT", "threshold": "BLOCK_NONE"},
+    ]
+    
+    response = model.generate_content(
+        full_context,
+        generation_config=generation_config,
+        safety_settings=safety_settings,
+        stream=True
+    )
+    
+    # Collect the response chunks
+    response_text = ""
+    for chunk in response:
+        response_text += chunk.text
+        yield [[message, response_text]]
+
+def create_interface():
+    """Create and configure the Gradio interface"""
+    with gr.Blocks(css="footer {visibility: hidden}") as demo:
+        gr.Markdown("# You are Asking Google Deep Mind about \"From Whispers to Voices\", it needs 15 seconds to think")
+        chatbot = gr.Chatbot(show_label=False)
+        
+        # Add custom CSS for wider chat window and proper scrolling
+        gr.HTML("""
+            <style>
+            .gradio-container {
+                max-width: 95% !important;
+                margin-left: auto !important;
+                margin-right: auto !important;
+                min-height: 100vh !important;
+            }
+            .contain {
+                min-height: 85vh !important;
+            }
+            .wrap.svelte-byatnx {
+                max-height: none !important;
+                overflow: visible !important;
+            }
+            .message.svelte-byatnx {
+                overflow-wrap: break-word !important;
+                white-space: pre-wrap !important;
+            }
+            </style>
+        """)
+        
+        # Auto-trigger the initial prompt on page load
+        def on_load():
+            response = process_initial_prompt()
+            response_text = ""
+            for chunk in response:
+                response_text += chunk.text
+                yield [[None, response_text]]
+        
+        demo.load(on_load, None, [chatbot])
+    
+    return demo
+
+# Create and launch the interface
+demo = create_interface()
+
+if __name__ == "__main__":
+    demo.launch()

instructions.txt

@@ -0,0 +1,480 @@
+1. We are going to build a very simple Gradio app or hugging face space that is going to start by passing a large prompt to the new Gemini 2.0 Flash thinking experimental 1-21 model and ask a specific question. 
+
+2. The intent will be that the user will have the answer streaming as soon as they open the app.  The user will be able to ask a new question using the context in the original prompt and the streaming thinking that is available in the API. 
+
+3. We will deploy this with a simple link to the gradio app and we want to start the app by automatically passing this prompt to the API and waiting for the reply: 
+
+This is the exact prompt we want to pass to the model but we also want to "hide" this from the user so the first thing the user sees as they open the gradio App is the reply to this question: 
+
+"based on the totality of your training, how revolutionary is this project by Christian H. Cooper given it is a working model that has been fine tuned and deployed 
+
+
+# From Whispers to Voices: A "Community-In-The-Loop" Proposal for Model Distillation and Language Preservation
+
+New Years Day, 2025
+
+A working model of the Stoney Nakoda language has been developed and is now available for community-in-the-loop testing in 2025:
+
+- **Model App**: [Stoney Language Model App](https://huggingface.co/spaces/HarleyCooper/StoneyApp)  
+- **Training Data**: [StoneyNakoda Training Dataset](https://huggingface.co/datasets/HarleyCooper/StoneyNakoda/blob/main/zSTONEY1_TRAINING_SET.jsonl)
+
+
+Any First Nations community seeking to apply this approach to their own language is warmly invited to reach out. 
+
+By following this code, you can build a model for any low-resource language. The starting dictionary size should be ~8,000 words.
+
+---
+
+## Table of Contents
+
+1. [New Years Day, Canadian Rockies, 2025](#introduction)  
+2. [Understanding How AI Learns Stoney Words Using Cosine Similarity](#understanding-how-ai-learns-stoney-words-using-cosine-similarity)
+3. [Project Architecture](#project-architecture)  
+   - [High-Level System Design](#high-level-system-design)  
+   - [Data Flow](#data-flow)  
+4. [Detailed Project Structure](#detailed-project-structure)  
+5. [Core Components](#core-components)  
+   - [Data Generation & Processing](#data-generation--processing)
+   - [Model Training](#model-training)
+6. [Comprehensive Setup Instructions](#comprehensive-setup-instructions)  
+   - [System Requirements](#system-requirements)
+   - [Environment Setup](#environment-setup)
+   - [Configuration](#configuration)
+   - [Initialization](#initialization)
+7. [Detailed Usage Pipeline](#detailed-usage-pipeline)  
+   1. [Generate Training Data](#1-generate-training-data)  
+   2. [Prepare Fine-tuning Data](#2-prepare-fine-tuning-data)  
+   3. [Fine-tune Model](#3-fine-tune-model)  
+8. [Advanced Model Configuration](#advanced-model-configuration)  
+   - [OpenAI Models](#openai-models)  
+   - [Google Gemini](#google-gemini)  
+   - [Hyperparameters](#hyperparameters)  
+9. [Comprehensive Data Formats](#comprehensive-data-formats)  
+   - [Dictionary Format](#dictionary-format)  
+   - [Q&A Format](#qa-format)  
+   - [OpenAI Training Format](#openai-training-format)  
+10. [Development Guidelines](#development-guidelines)  
+11. [Contributing](#contributing)  
+12. [License](#license)  
+13. [Acknowledgments](#acknowledgments)  
+14. [The Community-in-the-Loop Revolution](#the-community-in-the-loop-revolution)  
+    - [Introduction](#introduction-1)  
+    - [Conceptual Overview](#conceptual-overview)  
+    - [Heart of the Approach](#heart-of-the-approach)  
+    - [LoRA Fine-Tuning](#lora-fine-tuning)  
+    - [Mathematical Foundations](#mathematical-foundations)  
+    - [Mermaid Diagram](#mermaid-diagram)  
+    - [Cultural Integrity](#cultural-integrity)  
+    - [Data Sources](#data-sources)  
+    - [Expanding the Concept](#expanding-the-concept)  
+    - [Adaptive Checkpoints](#adaptive-checkpoints)  
+    - [Example Workflow](#example-workflow)  
+    - [Monitoring & QA](#monitoring--qa)  
+    - [Future Directions](#future-directions)  
+    - [Glossary](#glossary)  
+
+---
+
+## Introduction
+
+In my office, there is a murder; a map of one, at least.  
+
+![Dawson's Map of the Bow Valley](Public/FullDawsonMap.jpg)
+
+George Mercer Dawson explored the Bow Valley in the late 1800s, noting language on the British Columbia side. His map, though richly colored, stands like a tombstone over the Bow Valley where the Stoney people lived because he made no notes on their language and simply noted the people as "recent immigrants"
+
+![Detail of Dawson Map](Public/dawsondetail.jpg)
+
+What is very obvious from the linguistic patterns among the Haida, Tshimsia, Thlinkit, Kwakiool and Kawitshin dialects nearby is that languages blend like “linguistic DNA,” and machine learning could help trace faint threads of lost speech to their roots. Where some see isolation as a curse, in the age of AI, Stoney’s isolation turns out to be its strength.
+
+For about two years, I thought about the size of the vector space that would be needed to get a model to self-train on a set of 100% indigenous data, and how that model could refine its grasp of the broader Stoney Language. This is now publicly and freely available. 
+
+
+Two key releases influenced my thinking of what was possible:
+
+1. [Meta’s Llama-3 Model (April 18th, 2024)](https://www.reuters.com/technology/meta-releases-early-versions-its-llama-3-ai-model-2024-04-18/)  
+2. [OpenAI Fine-Tuning API (October 2024)](https://openai.com/index/api-model-distillation/)
+
+Both gave me the motivation to build what’s presented here. The true innovation here lies in how communities can narratively correct the initially flawed response (about 10% of the time, the model works every time.) then that feeback be passed seamleslly back into the fine-tuning process. The [textbooks](https://globalnews.ca/news/9430501/stoney-nakota-language-textbook/) that the Stoney community created—intended as educational tools—became perfect concept of a model prompts, each chapter or word offering pure indigenous data devoid of external weights or biases to the fine-tuning process.
+
+
+Early in 2023, I found an original, unpublished sketch by James Hector likely drawn in the summer of 1858 or 1859 along the Bow River in Southern Alberta:
+
+![Sketch by James Hector of a Stoney Woman](Public/StoneyWoman.jpg)
+
+Finding this, and already aware of George Mercer Dawson's work on First Nation's language on the British Columbia side, I was inspired to put the effort in and build a working model of the language and implement the Community-In-The-Loop distillation method.
+
+This sketch shifted my thinking from considering the "Stoney People” to this "Stoney Woman” who saw these same mountains and rivers I see everyday, yet who had a very different way to think about and communicate to the world around her.  The Community-in-the-Loop model distillation will quickly converge this initial model toward fluencey. I suspect this will require the community to correct about 80,000 question and answer pairs and would cost less than $800 in OpenAI computing power. Recent releases by Google and the Chinese Lab DeepSeek, could effectively reduce the cost to zero.  
+
+I think what this project has left me considering most ist that a century from now, strangers will live in all our homes and most of what we worry about today will not matter. But we can honor “Stoney Woman” by making sure her language endures, forging a living record in an age of AI. Incredibly, this tool will work with any first nations language, as long as there is a starting dictionary of about 8,000 words. 
+
+**I am freely available to help any First Nation in Canada.**
+
+## Understanding How AI Learns Stoney Words Using Cosine Similarity
+
+Word Embeddings: Mapping Words in Space
+Word embeddings are like placing words in a high-dimensional map, where similar words are positioned closer together. For example, "strawberry," "orange," and "cherry" might form a cluster because they are fruits, while "laptop," "Microsoft," and "Android" might cluster elsewhere as tech-related terms. Each axis in this space represents a characteristic of the words, such as their context or meaning.
+
+Context Shapes Meaning
+A word's position in this space isn't fixed—it shifts based on context. For instance, the word "apple" could mean a fruit or the tech brand, depending on its surrounding words, like "buy" (tech) or "tree" (fruit). This dynamic placement captures the nuances of meaning.
+
+Cosine Similarity: Measuring Relationships
+Cosine similarity quantifies how similar two words are by measuring the angle between their vectors in the embedding space:
+
+- Similar words have vectors pointing in nearly the same direction (cosine similarity close to 1)
+- Unrelated words have vectors at a right angle (cosine similarity near 0)
+- Opposite meanings have vectors pointing in opposite directions (cosine similarity close to -1)
+- For example, "cherry" and "orange" might have a similarity of 0.97, while "cherry" and "laptop" might score 0.24
+
+How AI Learns Stoney Words
+
+- **Stoney Dictionary as a Starting Point:**
+  The AI begins with a structured dictionary of Stoney words, including translations, categories, pronunciations, and cultural context.
+
+- **Community Feedback for Learning:**
+  The AI makes initial translations, which are often incorrect. Stoney speakers provide corrections, enriched with cultural context, stories, and humor. This feedback helps refine the AI's understanding.
+
+The Role of Cosine Similarity in AI Learning
+
+- The AI uses word embeddings to group Stoney words based on their meaning. For example, it determines whether a word belongs to a category like "fruit," "animal," or "spiritual."
+- Community corrections and cosine similarity guide the AI in repositioning words closer to their accurate groupings in the embedding space.
+
+Iterative Refinement
+Through repeated feedback and fine-tuning, the AI improves its ability to place Stoney words correctly, not just individually but in the context of sentences and paragraphs. Over time, it develops a detailed, dynamic map of the Stoney language, with words clustered according to their community-informed meanings and uses.
+
+Although this is not cosine similarity, you can see the relationships among words can concepts in Stoney as I have mapped them here: https://atlas.nomic.ai/data/harleycoops/stoney-nakoda-language-synthetic/map/5c87caaf-6be0-4546-9e83-826569070b24#nqlL
+
+
+---
+
+## Project Architecture
+
+This code forms a complete pipeline for training and deploying a Stoney model. It is fully functional—but not correct 100% of the time—and is designed to improve through Community-In-The-Loop feedback. Access the model here:  
+[Stoney Language Model App](https://huggingface.co/spaces/HarleyCooper/StoneyApp)
+
+### High-Level System Design
+
+1. **Data Ingestion Layer**  
+2. **Processing Pipeline** (Q&A generation, augmentation, conversion)  
+3. **Model Training Framework** (fine-tuning, hyperparameters, monitoring)  
+4. **Inference Interface** (API endpoint, response formatting, error handling)
+
+### Data Flow
+
+1. Raw dictionary data → Data Ingestion  
+2. Processed data → Q&A Generation  
+3. Generated Q&A pairs → Training Data Preparation  
+4. Prepared data → Model Fine-tuning  
+5. Fine-tuned model → Inference Interface  
+
+---
+
+## Detailed Project Structure
+
+```
+PUBLICRELEASE/
+├── OpenAIFineTune/           # OpenAI fine-tuning files
+│   ├── stoney_train.jsonl    # Training dataset
+│   └── stoney_valid.jsonl    # Validation dataset
+├── checkpoints/              # Model checkpoints
+├── .env.example             # Env variables example
+├── requirements.txt         # Python dependencies
+├── english_dictionary.jsonl
+├── stoney_dictionary.jsonl
+└── bilingual_training_set.jsonl
+```
+
+---
+
+## Core Components
+
+### Data Generation & Processing
+
+- **`bilingual_qa_generator.py`**  
+  Generates Q&A pairs from dictionaries, using advanced language generation.
+
+- **`convert_data_format.py`**  
+  Supports multiple data formats; validates and enforces schemas.
+
+- **`finetunesetup.py`**  
+  Splits data (80/20) with stratified sampling and prepares files.
+
+### Model Training
+
+- **`openai_finetune.py`**  
+  Handles fine-tuning, error handling, checkpointing, and logging.
+
+---
+
+## Comprehensive Setup Instructions
+
+### System Requirements
+
+- Python 3.8+  
+- 8GB+ RAM (16GB recommended)  
+- 10GB free disk space  
+- Stable internet connection  
+
+### Environment Setup
+
+```bash
+# Clone the repository
+git clone [repository-url]
+cd PUBLICRELEASE
+
+# Create and activate a virtual environment
+python -m venv venv
+source venv/bin/activate  # Windows: venv\Scripts\activate
+
+# Install dependencies
+pip install -r requirements.txt
+
+```
+
+### Configuration
+
+```bash
+# Copy example environment file
+cp .env.example .env
+# Provide OPENAI_API_KEY, GOOGLE_API_KEY in .env
+
+```
+
+### Initialization
+
+```bash
+python initialize.py
+
+```
+
+----------
+
+## Detailed Usage Pipeline
+
+### 1. Generate Training Data
+
+```bash
+python bilingual_qa_generator.py
+
+```
+
+-   Processes `english_dictionary.jsonl` & `stoney_dictionary.jsonl`
+-   Produces `bilingual_training_set.jsonl`
+
+### 2. Prepare Fine-tuning Data
+
+```bash
+python finetunesetup.py
+
+```
+
+-   Converts Q&A to OpenAI format
+-   Outputs `OpenAIFineTune/stoney_train.jsonl` & `stoney_valid.jsonl`
+
+### 3. Fine-tune Model
+
+```bash
+python openai_finetune.py
+
+```
+
+-   Uploads files to OpenAI
+-   Monitors fine-tuning progress
+-   Implements checkpointing & logs
+
+----------
+
+## Advanced Model Configuration
+
+### OpenAI Models
+
+-   Default: `gpt-4o-2024-08-06`
+-   Alternative: `gpt-3.5-turbo`
+-   `.env`: `OPENAI_MODEL`
+
+### Google Gemini
+
+-   Default: `gemini-2.0-exp`
+-   `.env`: `GEMINI_MODEL`
+
+### Hyperparameters
+
+-   LR: `1e-5`
+-   Batch size: `32`
+-   Epochs: `3`
+-   Context window: `4096`
+
+----------
+
+## Comprehensive Data Formats
+
+### Dictionary Format
+
+```json
+{
+  "english_word": "example",
+  "stoney_versions": [
+    {
+      "word": "...",
+      "grammatical_classification": "...",
+      "meaning": "..."
+    }
+  ]
+}
+
+```
+
+### Q&A Format
+
+```json
+{
+  "question": "How do you say X in Stoney?",
+  "answer": "The Stoney word for X is...",
+  "source_language": "english",
+  "generated_at": "timestamp"
+}
+
+```
+
+### OpenAI Training Format
+
+```json
+{
+  "messages": [
+    {"role": "system", "content": "You are a bilingual Stoney-English assistant..."},
+    {"role": "user", "content": "question"},
+    {"role": "assistant", "content": "answer"}
+  ]
+}
+
+```
+
+----------
+
+## Development Guidelines
+
+-   **Style**: PEP 8, type hints, docstrings, consistent naming
+-   **Testing**: Unit tests, integration tests, CI, coverage
+-   **Documentation**: Inline comments, usage examples, troubleshooting
+
+----------
+
+## Contributing
+
+1.  Fork, branch, implement changes, test
+2.  Submit a pull request
+
+**Code Review**
+
+-   Clear commits, small changes, documentation, test coverage
+
+----------
+
+## The Community-in-the-Loop Revolution
+
+### Introduction
+
+This project aims to preserve, refine, and resurrect endangered languages via AI fine-tuning and model distillation. Minimal lexical data can evolve into a culturally rich digital speaker of Stoney Nakoda. This subverts assumptions that massive datasets are necessary, instead emphasizing:
+
+-   Iterative improvement with community feedback
+-   Narrative corrections (cultural context over simple dictionary entries)
+-   Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning
+
+### Conceptual Overview
+
+**Community-in-the-Loop Model Distillation**:
+
+1.  Start with a small dictionary/text set.
+2.  Prompt an initial model.
+3.  Let the community correct errors with storytelling and context, not just words.
+4.  LoRA-based fine-tuning absorbs these narrative corrections.
+5.  The model evolves iteratively, guided by cultural custodians.
+
+### Heart of the Approach
+
+-   **Intentional Errors**: Poke the model with tough or context-specific queries.
+-   **Narrative Corrections**: Rich cultural commentary instead of bare “right vs. wrong.”
+-   **Distillation Triplets**: (Prompt, Disallowed Reply, Narrative Reply).
+-   **Iterative Improvement**: If the model stumbles, revert and add more context.
+
+### LoRA Fine-Tuning
+
+LoRA attaches small, low-rank matrices to the base model. This dramatically reduces compute and speeds up retraining:
+
+-   **Efficiency**: Fraction of resources required vs. full retraining
+-   **Focused Updates**: Capturing the “essence” of new knowledge
+-   **Rapid Iterations**: Frequent refinement without heavy overhead
+
+### Mathematical Foundations
+
+If W0\mathbf{W}_0 is the base weight matrix, LoRA introduces ΔW=AB\Delta \mathbf{W} = \mathbf{A}\mathbf{B} with A∈Rd×r\mathbf{A} \in \mathbb{R}^{d \times r} and B∈Rr×k\mathbf{B} \in \mathbb{R}^{r \times k}, where r≪min⁡(d,k)r \ll \min(d,k). Loss functions track both linguistic and cultural accuracy (e.g., a “Cultural Authenticity Score”).
+
+### Mermaid Diagram
+
+```mermaid
+graph TD
+    A[Initial Model] --> B[Generate Response]
+    B --> C{Correct?}
+    C -->|No| D[Community Correction]
+    D --> E[Create Distillation Triplet]
+    E --> F[LoRA Fine-Tuning]
+    F --> A
+    C -->|Yes| G[Validation]
+
+```
+
+### Cultural Integrity
+
+Every correction preserves cultural norms—idioms, humor, oral traditions—and ensures the community wields control over the AI’s “mindset.”
+
+### Data Sources
+
+A 10,000-word Stoney Nakoda dictionary and community textbooks serve as seeds. Community feedback enriches this data over time, weaving historical memory into the model.
+
+### Expanding the Concept
+
+From a tiny dictionary to an AI that:
+
+-   **Understands context** (formal/informal usage)
+-   **Integrates cultural references** (stories, metaphors)
+-   **Remembers history** (ancestors, ceremonies, seasonal events)
+
+### Adaptive Checkpoints
+
+-   **Forward Progress**: Keep the new checkpoint if improved.
+-   **Reversion**: If degraded, roll back and increase context in corrections.
+-   **Convergence**: Repeat until stable authenticity and fluency metrics are met.
+
+### Example Workflow
+
+1.  **Prompt**: “How to say ‘taste slightly with the tip of your tongue’ in Stoney?”
+2.  **Model’s Flawed Reply**: “`supthîyach`” (incorrect).
+3.  **Community Correction**: Shares the correct phrase plus a story from childhood.
+4.  **Distillation Triplet**: (Prompt, Disallowed, Narrative).
+5.  **LoRA Fine-Tuning**: Model adjusts swiftly.
+6.  **Re-Evaluation**: Answers improve in subsequent queries.
+
+### Monitoring & QA
+
+-   **Cultural Authenticity Score (CAS)**
+-   **Linguistic Fluency** (perplexity, cross-entropy)
+-   **Validation Loops** (watch for regressions, revert if needed)
+
+### Future Directions
+
+-   **Oral Histories**: Model retells century-old stories.
+-   **Seasonal Knowledge**: Terms tied to ceremonies and ecological cycles.
+-   **Dialects/Accents**: Respecting sub-regional differences.
+-   **Educational Tools**: Interactive AI for language learning.
+-   **Ethical AI**: Centered on consent, community governance, cultural integrity.
+
+### Glossary
+
+-   **CAS**: Cultural Authenticity Score
+-   **Distillation Triplet**: (Prompt, Flawed Reply, Narrative Reply)
+-   **LoRA**: Low-Rank Adaptation
+-   **Community-in-the-Loop**: Paradigm of continuous human-guided refinement
+
+
+
+"

requirements.txt

@@ -0,0 +1,43 @@
+dropbox
+gradio
+mistralai
+anthropic
+pinecone-client
+nomic
+openai
+groq
+langchain-community
+replicate
+google-generativeai
+perplexityai
+cohere
+langchain
+alpha_vantage
+huggingface_hub
+tavily-python
+python-mathpix
+requests
+python-dotenv
+sympy
+numpy-financial
+numpy
+pandas
+mesop
+langchain-anthropic
+langchain-openai
+openpyxl
+beautifulsoup4
+langchain-mistralai
+google-cloud-secret-manager
+selenium
+google-auth-oauthlib
+google-auth-httplib2
+google-api-python-client
+PyPDF2
+python-docx
+markdown
+Pygithub
+GitPython
+SpeechRecognition
+google-cloud-storage
+nltk

tools.py

@@ -0,0 +1,864 @@
+'''
+Guidelines for Creating and Utilizing Tools in tools.py:
+
+1. Initial Assessment:
+
+Review Existing Tools:
+Before adding new functions, thoroughly read through tools.py to understand the existing tools and their functionalities.
+Determine if an existing tool can be adapted or extended to meet the current needs, avoiding redundancy.
+2. Tool Creation and Function Design:
+
+Create Within tools.py:
+
+Add new tools as functions within tools.py. If tools.py doesn't exist, create it.
+Ensure each function is self-contained and focused on a single task for modularity.
+Design for Importing:
+
+Design tools to be imported and executed via terminal commands. Do not include execution code that runs when tools.py is imported.
+Follow Best Practices:
+
+Use clear and descriptive function names that reflect their general purpose.
+Include docstrings for each function, detailing the purpose, parameters, and expected outputs.
+Adhere to PEP 8 style guidelines for readable and maintainable code.
+3. Generalization:
+
+Broad Input Handling:
+
+Design functions to handle a wide range of inputs, enhancing reusability for future tasks.
+Accept parameters that allow the function to be applicable in various scenarios (e.g., any stock ticker, URL, or data file).
+Flexible Functionality:
+
+Ensure functions can process different data types and structures when applicable.
+Avoid hardcoding values; use parameters and defaults where necessary.
+Modularity:
+
+If a task involves multiple distinct operations, split them into separate functions.
+This approach enhances clarity and allows for individual functions to be reused independently.
+4. Execution and Script Management:
+
+Import and Run via Terminal:
+
+Do not execute tools.py directly. Instead, import the necessary functions and run them using the terminal.
+Use the command:
+bash
+Copy code
+python -c "from tools import function_name; function_name(args)"
+Replace function_name and args with the appropriate function and arguments.
+Avoid Additional Scripts:
+
+Do not create extra .py files or scripts for execution purposes.
+Keep all tool functions within tools.py and execute them using the import method shown above.
+5. Output:
+
+Console Printing:
+
+Ensure that all tools print their output directly to the console.
+Format the output for readability, using clear messages and organizing data in a logical manner.
+No Return Statements for Output:
+
+While functions can return values for internal use, the primary results should be displayed using print() statements.
+6. Error Handling:
+
+Input Validation:
+
+Validate all input parameters to catch errors before execution.
+Provide informative error messages to guide correct usage.
+Exception Management:
+
+Use try-except blocks to handle potential exceptions without crashing the program.
+Log errors where appropriate, and ensure they don't expose sensitive information.
+Debugging and Testing:
+
+Test functions with various inputs, including edge cases, to ensure robustness.
+If errors are found, revise and debug the functions promptly.
+7. Post-Creation:
+
+Execute to Fulfill Requests:
+
+After creating or updating tools, execute them as needed to fulfill user requests unless the request was solely for tool creation.
+Documentation:
+
+Update any relevant documentation or comments within tools.py to reflect new additions or changes.
+Consider maintaining a usage example within the docstring for complex functions.
+8. Maintenance and Refactoring:
+
+Regular Review:
+
+Periodically review tools.py to identify opportunities for optimization and improvement.
+Remove or update deprecated functions that are no longer effective or necessary.
+Enhance Generalization:
+
+Refactor functions to improve their general applicability as new requirements emerge.
+Stay vigilant for patterns that can be abstracted into more general solutions.
+9. Compliance and Security:
+
+Data Protection:
+
+Ensure that tools handle data securely, especially when dealing with sensitive information.
+Avoid hardcoding credentials or exposing private data through outputs.
+Licensing and Dependencies:
+
+Verify that any third-party libraries used are properly licensed and documented.
+Include installation instructions for dependencies if they are not part of the standard library.
+'''
+
+# Your tools will be defined below this line
+
+import os
+from datetime import datetime, timedelta
+import PyPDF2
+from pathlib import Path
+import json
+import re
+import requests
+from typing import List, Dict, Optional, Union
+import subprocess
+import sys
+from git import Repo
+
+def get_current_month_folder() -> str:
+    """Returns the path to the current month's folder."""
+    base_path = r"C:\Users\admin\Dropbox\Current\2024"
+    current_month = datetime.now().strftime("%B")  # Full month name
+    return os.path.join(base_path, current_month)
+
+def get_pdfs_for_date(target_date: datetime = None) -> List[str]:
+    """
+    Finds all PDFs saved on a specific date in the current month's folder structure.
+    Args:
+        target_date: datetime object for the target date. If None, uses today's date.
+    Returns a list of full paths to PDF files.
+    """
+    if target_date is None:
+        target_date = datetime.now()
+    
+    target_date_str = target_date.strftime("%Y-%m-%d")
+    month_folder = get_current_month_folder()
+    pdf_files = []
+    
+    # Walk through all subdirectories
+    for root, _, files in os.walk(month_folder):
+        for file in files:
+            if file.lower().endswith('.pdf'):
+                file_path = os.path.join(root, file)
+                # Get file's modification time
+                mod_time = datetime.fromtimestamp(os.path.getmtime(file_path))
+                if mod_time.strftime("%Y-%m-%d") == target_date_str:
+                    pdf_files.append(file_path)
+    
+    return pdf_files
+
+def extract_text_from_pdf(pdf_path: str) -> str:
+    """Extract text content from a PDF file."""
+    try:
+        with open(pdf_path, 'rb') as file:
+            reader = PyPDF2.PdfReader(file)
+            text = ""
+            for page in reader.pages:
+                text += page.extract_text() + "\n"
+            return text
+    except Exception as e:
+        print(f"Error processing {pdf_path}: {str(e)}")
+        return ""
+
+def summarize_pdfs_for_date(target_date: datetime = None):
+    """
+    Main function to process and summarize PDFs for a specific date.
+    Args:
+        target_date: datetime object for the target date. If None, uses today's date.
+    Prints summary to console and saves to a JSON file.
+    """
+    if target_date is None:
+        target_date = datetime.now()
+    
+    pdfs = get_pdfs_for_date(target_date)
+    if not pdfs:
+        print(f"No PDFs found for {target_date.strftime('%Y-%m-%d')}")
+        return
+
+    summaries = {}
+    for pdf_path in pdfs:
+        print(f"Processing: {pdf_path}")
+        text = extract_text_from_pdf(pdf_path)
+        
+        # Basic summary: first 500 characters of text
+        summary = text[:500] + "..." if len(text) > 500 else text
+        
+        # Store in dictionary with filename as key
+        filename = os.path.basename(pdf_path)
+        summaries[filename] = {
+            "path": pdf_path,
+            "summary": summary,
+            "processed_date": datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+        }
+    
+    # Save summaries to JSON file in the current directory
+    output_dir = "summaries"
+    os.makedirs(output_dir, exist_ok=True)
+    
+    output_file = os.path.join(output_dir, f"summaries_{target_date.strftime('%Y-%m-%d')}.json")
+    with open(output_file, 'w', encoding='utf-8') as f:
+        json.dump(summaries, f, indent=4, ensure_ascii=False)
+    
+    print(f"\nProcessed {len(pdfs)} PDFs")
+    print(f"Summaries saved to: {output_file}")
+    
+    # Print summaries to console
+    for filename, data in summaries.items():
+        print(f"\n{'='*80}\n{filename}")
+        print(f"Path: {data['path']}")
+        print(f"\nSummary:\n{data['summary'][:200]}...")
+
+class WebSearchTool:
+    """
+    A tool for performing web searches using the Perplexity API.
+    Designed to be the default web search mechanism for context-requiring queries.
+    """
+    
+    def __init__(self, api_key: Optional[str] = None):
+        """
+        Initialize the WebSearchTool.
+        
+        Args:
+            api_key: Perplexity API key. If None, will try to get from environment variable.
+        """
+        self.api_key = api_key or os.getenv("PERPLEXITY_API_KEY")
+        if not self.api_key:
+            raise ValueError("Perplexity API key must be provided or set in PERPLEXITY_API_KEY environment variable")
+        
+        self.headers = {
+            "Authorization": f"Bearer {self.api_key}",
+            "Content-Type": "application/json"
+        }
+    
+    def search(self, query: str, max_results: int = 5) -> Dict:
+        """
+        Perform a web search using Perplexity API.
+        
+        Args:
+            query: The search query
+            max_results: Maximum number of results to return
+            
+        Returns:
+            Dictionary containing search results and metadata
+        """
+        try:
+            # Make the API request
+            response = requests.post(
+                "https://api.perplexity.ai/chat/completions",
+                headers=self.headers,
+                json={
+                    "model": "llama-3.1-sonar-huge-128k-online",
+                    "messages": [
+                        {
+                            "role": "system",
+                            "content": "You are a helpful assistant that provides accurate and concise answers based on web search results."
+                        },
+                        {
+                            "role": "user",
+                            "content": query
+                        }
+                    ]
+                }
+            )
+            response.raise_for_status()
+            data = response.json()
+            
+            # Extract answer from the response
+            answer = data.get("choices", [{}])[0].get("message", {}).get("content", "No answer found")
+            
+            # Process and format the results
+            results = {
+                "query": query,
+                "timestamp": datetime.now().isoformat(),
+                "answer": answer,
+                "references": [],  # References not available in this API version
+                "metadata": {
+                    "source": "Perplexity API",
+                    "model": "llama-3.1-sonar-huge-128k-online"
+                }
+            }
+            return results
+            
+        except Exception as e:
+            print(f"Error performing search: {str(e)}")
+            return {
+                "query": query,
+                "timestamp": datetime.now().isoformat(),
+                "error": str(e),
+                "metadata": {
+                    "source": "Perplexity API",
+                    "status": "error"
+                }
+            }
+    
+    def format_results(self, results: Dict, format: str = "text") -> str:
+        """
+        Format search results in the specified format.
+        
+        Args:
+            results: Search results dictionary
+            format: Output format ("text" or "markdown")
+            
+        Returns:
+            Formatted string of results
+        """
+        if "error" in results:
+            return f"Error: {results['error']}"
+            
+        if format == "markdown":
+            output = f"# Search Results for: {results['query']}\n\n"
+            output += f"## Answer\n{results['answer']}\n\n"
+            if results['references']:
+                output += "## References\n"
+                for i, ref in enumerate(results['references'], 1):
+                    output += f"{i}. {ref['title']} - {ref['url']}\n"
+            return output
+        else:
+            output = f"Search Results for: {results['query']}\n\n"
+            output += f"Answer:\n{results['answer']}\n\n"
+            if results['references']:
+                output += "References:\n"
+                for i, ref in enumerate(results['references'], 1):
+                    output += f"{i}. {ref['title']} - {ref['url']}\n"
+            return output
+
+class MCPServerManager:
+    """
+    A tool for installing and managing Model Context Protocol (MCP) servers.
+    Integrates with Claude desktop and manages server configurations.
+    """
+    
+    DEFAULT_CONFIG_LOCATIONS = [
+        "mcp.json",
+        ".mcp/config.json",
+        "config/mcp.json",
+        "mcp_config.json",
+        ".config/mcp/servers.json"
+    ]
+    
+    def __init__(self, base_dir: Optional[str] = None):
+        """
+        Initialize the MCP Server Manager.
+        
+        Args:
+            base_dir: Base directory for installing servers. If None, uses current directory.
+        """
+        self.base_dir = Path(base_dir) if base_dir else Path.cwd() / "mcp_servers"
+        self.base_dir.mkdir(parents=True, exist_ok=True)
+        self.servers_repo_url = "https://github.com/modelcontextprotocol/servers.git"
+        self.installed_servers = {}
+        self.load_installed_servers()
+    
+    def load_installed_servers(self):
+        """Load information about installed servers from the config file."""
+        config_file = self.base_dir / "config.json"
+        if config_file.exists():
+            with open(config_file, "r") as f:
+                self.installed_servers = json.load(f)
+    
+    def save_installed_servers(self):
+        """Save information about installed servers to the config file."""
+        config_file = self.base_dir / "config.json"
+        with open(config_file, "w") as f:
+            json.dump(self.installed_servers, f, indent=4)
+    
+    def get_featured_servers(self) -> List[Dict]:
+        """
+        Get list of featured servers from the MCP GitHub repository.
+        
+        Returns:
+            List of server information dictionaries
+        """
+        try:
+            # Clone or update the servers repository
+            repo_dir = self.base_dir / "servers_repo"
+            if repo_dir.exists():
+                repo = Repo(repo_dir)
+                repo.remotes.origin.pull()
+            else:
+                repo = Repo.clone_from(self.servers_repo_url, repo_dir)
+            
+            # First try to read from featured.json
+            featured_file = repo_dir / "featured.json"
+            if featured_file.exists():
+                with open(featured_file, "r", encoding="utf-8") as f:
+                    return json.load(f)
+            
+            # If featured.json doesn't exist, parse README.md
+            readme_file = repo_dir / "README.md"
+            if readme_file.exists():
+                servers = []
+                with open(readme_file, "r", encoding="utf-8", errors="ignore") as f:
+                    content = f.read()
+                    # Look for server repository links
+                    repo_links = re.findall(r"\[([^\]]+)\]\((https://github.com/[^)]+)\)", content)
+                    for name, url in repo_links:
+                        if "/modelcontextprotocol/" in url:
+                            servers.append({
+                                "name": name,
+                                "repository": url,
+                                "config": {}
+                            })
+                return servers
+            
+            return []
+            
+        except Exception as e:
+            print(f"Error getting featured servers: {str(e)}")
+            return []
+    
+    def find_server_config(self, server_dir: Path) -> Optional[Dict]:
+        """
+        Search for MCP server configuration in common locations.
+        
+        Args:
+            server_dir: Directory to search in
+            
+        Returns:
+            Server configuration dictionary if found, None otherwise
+        """
+        # First check common config file locations
+        for config_path in self.DEFAULT_CONFIG_LOCATIONS:
+            config_file = server_dir / config_path
+            if config_file.exists():
+                try:
+                    with open(config_file, "r", encoding="utf-8") as f:
+                        config = json.load(f)
+                        if "mcpServers" in config:
+                            return config["mcpServers"]
+                except Exception as e:
+                    print(f"Error reading config from {config_file}: {str(e)}")
+        
+        # Check package.json for Node.js projects
+        package_json = server_dir / "package.json"
+        if package_json.exists():
+            try:
+                with open(package_json, "r", encoding="utf-8") as f:
+                    config = json.load(f)
+                    if "mcpServers" in config:
+                        return config["mcpServers"]
+            except Exception as e:
+                print(f"Error reading config from package.json: {str(e)}")
+        
+        # Check pyproject.toml for Python projects
+        pyproject_toml = server_dir / "pyproject.toml"
+        if pyproject_toml.exists():
+            try:
+                import tomli
+                with open(pyproject_toml, "rb") as f:
+                    config = tomli.load(f)
+                    if "tool" in config and "mcp" in config["tool"]:
+                        return {"python": config["tool"]["mcp"]}
+            except ImportError:
+                print("tomli package not found, skipping pyproject.toml parsing")
+            except Exception as e:
+                print(f"Error reading config from pyproject.toml: {str(e)}")
+        
+        return None
+
+    def install_server(self, server_name: str, custom_config: Optional[Dict] = None) -> bool:
+        """
+        Install a specific MCP server.
+        
+        Args:
+            server_name: Name of the server to install
+            custom_config: Optional custom configuration for the server
+            
+        Returns:
+            True if installation was successful, False otherwise
+        """
+        try:
+            # Get server information from featured servers
+            featured_servers = self.get_featured_servers()
+            server_info = next((s for s in featured_servers if s["name"] == server_name), None)
+            if not server_info:
+                print(f"Server '{server_name}' not found in featured servers")
+                return False
+            
+            # Create server directory
+            server_dir = self.base_dir / server_name
+            server_dir.mkdir(exist_ok=True)
+            
+            # Clone server repository
+            repo = Repo.clone_from(server_info["repository"], server_dir)
+            
+            # Install dependencies
+            if (server_dir / "requirements.txt").exists():
+                subprocess.run([sys.executable, "-m", "pip", "install", "-r", "requirements.txt"], cwd=server_dir)
+            elif (server_dir / "package.json").exists():
+                subprocess.run(["npm", "install"], cwd=server_dir)
+            
+            # Find or use custom server configuration
+            config = custom_config or self.find_server_config(server_dir) or {}
+            
+            # Store server information
+            self.installed_servers[server_name] = {
+                "path": str(server_dir),
+                "version": repo.head.commit.hexsha[:8],
+                "install_date": datetime.now().isoformat(),
+                "config": config
+            }
+            self.save_installed_servers()
+            
+            print(f"Successfully installed {server_name}")
+            if config:
+                print(f"Found server configuration: {json.dumps(config, indent=2)}")
+            else:
+                print("No server configuration found. You may need to configure it manually.")
+            
+            return True
+            
+        except Exception as e:
+            print(f"Error installing server '{server_name}': {str(e)}")
+            return False
+    
+    def uninstall_server(self, server_name: str) -> bool:
+        """
+        Uninstall a specific MCP server.
+        
+        Args:
+            server_name: Name of the server to uninstall
+            
+        Returns:
+            True if uninstallation was successful, False otherwise
+        """
+        try:
+            if server_name not in self.installed_servers:
+                print(f"Server '{server_name}' is not installed")
+                return False
+            
+            # Remove server directory
+            server_dir = Path(self.installed_servers[server_name]["path"])
+            shutil.rmtree(server_dir)
+            
+            # Remove from installed servers
+            del self.installed_servers[server_name]
+            self.save_installed_servers()
+            
+            print(f"Successfully uninstalled {server_name}")
+            return True
+            
+        except Exception as e:
+            print(f"Error uninstalling server '{server_name}': {str(e)}")
+            return False
+    
+    def list_installed_servers(self) -> Dict[str, Dict]:
+        """
+        Get information about installed servers.
+        
+        Returns:
+            Dictionary of installed server information
+        """
+        return self.installed_servers
+    
+    def update_server(self, server_name: str) -> bool:
+        """
+        Update a specific MCP server to the latest version.
+        
+        Args:
+            server_name: Name of the server to update
+            
+        Returns:
+            True if update was successful, False otherwise
+        """
+        try:
+            if server_name not in self.installed_servers:
+                print(f"Server '{server_name}' is not installed")
+                return False
+            
+            server_dir = Path(self.installed_servers[server_name]["path"])
+            repo = Repo(server_dir)
+            
+            # Get current version
+            old_version = repo.head.commit.hexsha[:8]
+            
+            # Pull latest changes
+            repo.remotes.origin.pull()
+            
+            # Get new version
+            new_version = repo.head.commit.hexsha[:8]
+            
+            # Update dependencies if needed
+            if (server_dir / "requirements.txt").exists():
+                subprocess.run([sys.executable, "-m", "pip", "install", "-r", "requirements.txt"], cwd=server_dir)
+            
+            # Update stored information
+            self.installed_servers[server_name]["version"] = new_version
+            self.save_installed_servers()
+            
+            print(f"Updated {server_name} from {old_version} to {new_version}")
+            return True
+            
+        except Exception as e:
+            print(f"Error updating server '{server_name}': {str(e)}")
+            return False
+    
+    def configure_server(self, server_name: str, config: Dict) -> bool:
+        """
+        Configure a specific MCP server.
+        
+        Args:
+            server_name: Name of the server to configure
+            config: Configuration dictionary in the format:
+                   {
+                     "command": str,  # Command to run the server
+                     "args": List[str],  # Arguments for the command
+                     "env": Dict[str, str],  # Optional environment variables
+                     "cwd": str,  # Optional working directory
+                   }
+            
+        Returns:
+            True if configuration was successful, False otherwise
+        """
+        try:
+            if server_name not in self.installed_servers:
+                print(f"Server '{server_name}' is not installed")
+                return False
+            
+            server_dir = Path(self.installed_servers[server_name]["path"])
+            
+            # Validate configuration
+            if "command" not in config:
+                print("Error: Server configuration must include 'command'")
+                return False
+            
+            # Update configuration
+            self.installed_servers[server_name]["config"] = config
+            self.save_installed_servers()
+            
+            # Try to write configuration to a standard location
+            config_dir = server_dir / ".mcp"
+            config_dir.mkdir(exist_ok=True)
+            config_file = config_dir / "config.json"
+            
+            with open(config_file, "w", encoding="utf-8") as f:
+                json.dump({"mcpServers": {server_name: config}}, f, indent=2)
+            
+            print(f"Successfully configured {server_name}")
+            print(f"Configuration saved to {config_file}")
+            return True
+            
+        except Exception as e:
+            print(f"Error configuring server '{server_name}': {str(e)}")
+            return False
+
+    def start_server(self, server_name: str) -> bool:
+        """
+        Start a specific MCP server.
+        
+        Args:
+            server_name: Name of the server to start
+            
+        Returns:
+            True if server was started successfully, False otherwise
+        """
+        try:
+            if server_name not in self.installed_servers:
+                print(f"Server '{server_name}' is not installed")
+                return False
+            
+            server_info = self.installed_servers[server_name]
+            config = server_info.get("config", {})
+            
+            if not config:
+                print(f"Server '{server_name}' is not configured")
+                return False
+            
+            # Prepare command and arguments
+            command = config.get("command")
+            args = config.get("args", [])
+            env = {**os.environ, **(config.get("env", {}))}
+            cwd = config.get("cwd") or server_info["path"]
+            
+            # Start the server process
+            process = subprocess.Popen(
+                [command, *args],
+                env=env,
+                cwd=cwd,
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
+                text=True
+            )
+            
+            # Store process information
+            self.installed_servers[server_name]["process"] = process
+            self.save_installed_servers()
+            
+            print(f"Started server '{server_name}' (PID: {process.pid})")
+            return True
+            
+        except Exception as e:
+            print(f"Error starting server '{server_name}': {str(e)}")
+            return False
+
+    def stop_server(self, server_name: str) -> bool:
+        """
+        Stop a specific MCP server.
+        
+        Args:
+            server_name: Name of the server to stop
+            
+        Returns:
+            True if server was stopped successfully, False otherwise
+        """
+        try:
+            if server_name not in self.installed_servers:
+                print(f"Server '{server_name}' is not installed")
+                return False
+            
+            server_info = self.installed_servers[server_name]
+            process = server_info.get("process")
+            
+            if not process:
+                print(f"Server '{server_name}' is not running")
+                return False
+            
+            # Try to stop the process gracefully
+            process.terminate()
+            try:
+                process.wait(timeout=5)
+            except subprocess.TimeoutExpired:
+                process.kill()
+            
+            # Remove process information
+            del self.installed_servers[server_name]["process"]
+            self.save_installed_servers()
+            
+            print(f"Stopped server '{server_name}'")
+            return True
+            
+        except Exception as e:
+            print(f"Error stopping server '{server_name}': {str(e)}")
+            return False
+
+async def generate_pyflowchart(repo_path: str, output_dir: Optional[str] = None) -> None:
+    """
+    Generate flowcharts for all Python files in a repository using pyflowchart.
+    Creates HTML flowcharts that can be viewed in a browser.
+    
+    Args:
+        repo_path: Path to the repository
+        output_dir: Optional directory to save flowcharts. If None, creates a 'flowcharts' directory in the repo.
+    """
+    try:
+        # Ensure pyflowchart is installed
+        subprocess.run([sys.executable, "-m", "pip", "install", "pyflowchart"], check=True)
+        
+        # Set up output directory
+        if output_dir is None:
+            output_dir = os.path.join(repo_path, 'flowcharts')
+        os.makedirs(output_dir, exist_ok=True)
+        
+        # Generate HTML flowcharts
+        for root, _, files in os.walk(repo_path):
+            for file in files:
+                if file.endswith('.py'):
+                    py_file = os.path.join(root, file)
+                    
+                    # Skip empty files
+                    if os.path.getsize(py_file) == 0:
+                        print(f"Skipping empty file: {py_file}")
+                        continue
+                        
+                    # Check if file has actual Python code
+                    with open(py_file, 'r', encoding='utf-8') as f:
+                        content = f.read().strip()
+                        if not content:
+                            print(f"Skipping empty file: {py_file}")
+                            continue
+                    
+                    html_file = os.path.join(output_dir, f"{os.path.splitext(file)[0]}_flowchart.html")
+                    
+                    # Generate flowchart HTML
+                    print(f"Generating flowchart for {py_file}")
+                    try:
+                        subprocess.run([
+                            sys.executable, "-m", "pyflowchart", py_file, 
+                            "--output", html_file
+                        ], check=True)
+                        print(f"Saved HTML to {html_file}")
+                    except subprocess.CalledProcessError as e:
+                        print(f"Error generating flowchart for {py_file}: {str(e)}")
+                        continue
+        
+        print(f"\nFlowcharts generated in: {output_dir}")
+        
+    except subprocess.CalledProcessError as e:
+        print(f"Error running pyflowchart: {str(e)}")
+    except Exception as e:
+        print(f"Error generating flowcharts: {str(e)}")
+
+def extract_repo_context(repo_url: str, output_file: Optional[str] = None) -> None:
+    """
+    Extract repository context using gitingest.com.
+    
+    Args:
+        repo_url: GitHub repository URL
+        output_file: Optional file to save the context. If None, uses repo name with .txt extension.
+    """
+    try:
+        # Convert github.com URL to gitingest.com
+        if "github.com" not in repo_url:
+            raise ValueError("Only GitHub repositories are supported")
+            
+        ingest_url = repo_url.replace("github.com", "gitingest.com")
+        print(f"Fetching repository context from: {ingest_url}")
+        
+        # Make request to gitingest.com
+        response = requests.get(ingest_url)
+        response.raise_for_status()
+        
+        # Extract content
+        content = response.text
+        
+        # Save to file
+        if output_file is None:
+            repo_name = repo_url.split('/')[-1].replace('.git', '')
+            output_file = f"{repo_name}_context.txt"
+            
+        with open(output_file, 'w', encoding='utf-8') as f:
+            f.write(content)
+            
+        print(f"Repository context saved to: {output_file}")
+        
+    except requests.RequestException as e:
+        print(f"Error fetching repository context: {str(e)}")
+    except Exception as e:
+        print(f"Error extracting repository context: {str(e)}")
+
+async def post_clone_actions(repo_url: str) -> None:
+    """
+    Perform post-clone actions after a git clone operation:
+    1. Generate flowcharts for all Python files using pyflowchart
+    2. Extract repository context using gitingest.com
+    
+    Args:
+        repo_url: URL of the repository that was just cloned
+    """
+    try:
+        # Get the repository name from the URL
+        repo_name = repo_url.split('/')[-1].replace('.git', '')
+        repo_path = os.path.join(os.getcwd(), repo_name)
+        
+        # Generate flowcharts
+        print("\nGenerating flowcharts...")
+        await generate_pyflowchart(repo_path)
+        
+        # Extract repository context
+        print("\nExtracting repository context...")
+        extract_repo_context(repo_url)
+    
+    except Exception as e:
+        print(f"Error in post-clone actions: {str(e)}")
+
+# Example usage:
+if __name__ == "__main__":
+    # Initialize the search tool
+    search_tool = WebSearchTool()
+    
+    # Perform a simple search
+    results = search_tool.search("Latest developments in AI technology")
+    
+    # Print formatted results
+    print(search_tool.format_results(results, "markdown"))