Update README.md

README.md

@@ -1,40 +1,47 @@
 ---
 tags:
 - generated_from_trainer
+- finance
+- text-generation-inference
 model-index:
 - name: completed-model
   results: []
+language:
+- en
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
 [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
-# completed-model
-
-This model was trained from scratch on the None dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.3186
-- Rewards/chosen: -0.6296
-- Rewards/rejected: -2.5591
-- Rewards/accuracies: 0.8571
-- Rewards/margins: 1.9295
-- Logps/rejected: -296.3221
-- Logps/chosen: -425.5087
-- Logits/rejected: -2.2481
-- Logits/chosen: -1.7413
+
+Albatross is a collection of domain-specific language models for finance applications developed by [Gradient AI](https://gradient.ai/).
+
+This is the repository for an initial, demonstration version, the `v-alpha-tross`.
+
 
 ## Model description
 
-More information needed
+The `v-alpha-tross` is based on [meta-llama/Llama-2-70b-hf](https://huggingface.co/meta-llama/Llama-2-70b-hf), with additional, finance specific, pre-training, fine-tuning and instruction tuning.
+
+This model outperforms all other tuned Llama2-70B models on the Open LLM Leaderboard on Average score, and GSM8K. It is also skilled at extracting information from tabular data like those found in SEC filings.
+
 
 ## Intended uses & limitations
 
-More information needed
+The `v-alpha-tross` is intended as a demonstration of Gradient AI's Albatross framework for developing large language models specific to the finance domain. We welcome additional research and development, but do not plan on continued internal development on this legacy model.
+
+To get the expected performance, follow specific formatting requirements, including `INST` and `<<SYS>>` tags, and `<s>` tokens.
+
+## Training Strategy
+
+We aim to overcome deficiencies in general-purpose language models when faced with solving finance specific tasks.
+
+### Pre-Training
+
+A base Llama2-70B is further pre-trained on finance specific data since LLMs are poor at answering questions when their internal relevant document store is sparse [1].
+
+To curate enough quality training data with low operational overhead we use a novel data gathering approach. First, we crawl public repositories of text data. Second we adapt a LiRA membership inference technique to filter our crawled corpus to datasets not likely to be already in the base model's training. And third, we consult human professionals to review the (much smaller) filtered corpus to further remove low quality results. Lastly we build synthetic data pipelines to better represent under-sampled data, and train the model to better cope with variation in data representation and formatting.
 
-## Training and evaluation data
+<span style="color:red">[TODO: ADD MARKDOWN TABLE ]</span>
 
-More information needed
 
 ## Training procedure
 
@@ -75,4 +82,4 @@ The following hyperparameters were used during training:
 - Transformers 4.35.1
 - Pytorch 2.0.1+cu118
 - Datasets 2.14.7
-- Tokenizers 0.14.1
+- Tokenizers 0.14.1