Improve eval mainly

README.md

@@ -19,7 +19,7 @@ Krikri is built on top of [Llama-3.1-8B](https://huggingface.co/meta-llama/Llama
 # Model Information
 
 - Vocabulary extension of the Llama-3.1 tokenizer with Greek tokens
-- 128k context length (approximately 80,000 Greek words)
+- 128k context length (**approximately 80,000 Greek words**)
 - We extend the pretraining of Llama-3.1-8B with added proficiency for the Greek language, by utilizing a large training corpus. 
   * This corpus includes 56.7 billion monolingual Greek tokens, constructed from publicly available resources.
   * Additionaly, to mitigate catastrophic forgetting and ensure that the model has bilingual capabilities, we use additional sub-corpora with monolingual English texts (21 billion tokens) and Greek-English parallel data (5.5 billion tokens).
@@ -41,6 +41,7 @@ Chosen subsets of the 91 billion corpus were upsampled resulting in a size of **
 
 # How to use
 
+## With Transformers
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -58,7 +59,7 @@ outputs = model.generate(input_text['input_ids'], max_new_tokens=256, do_sample=
 print(tokenizer.batch_decode(outputs)[0])
 ```
 
-# How to serve with OpenAI compatible server via vLLM
+## With OpenAI compatible server via vLLM
 
 ```bash
 vllm serve ilsp/Llama-Krikri-8B-Base \
@@ -86,8 +87,14 @@ print(response.choices[0].text)
 
 # Evaluation
 
+Below, we report improvements of Llama-Krikri-8B-Base over Llama-3.1-8B for Greek and English:
+- **+10.8%** on Greek benchmarks
+- **+0.8%** on English benchmarks
+
+Our evaluations for Llama-Krikri-8B-Base, Llama-3.1-8B, and Meltemi 7B v1.5 are performed in a few-shot setting, consistent with the settings in the [Open LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). 
+
+## Greek Benchmarks
 
-## Greek Benchmarks 
 
 The evaluation suite we created for the Greek language includes 6 test sets. You can run the suite by cloning this [lighteval fork](https://github.com/LeonVouk/lighteval).
 
@@ -96,7 +103,7 @@ Our evaluation suite includes:
 * An existing benchmark for question answering in Greek ([Belebele](https://arxiv.org/abs/2308.16884))
 * A novel benchmark created by the ILSP team for medical question answering based on the medical exams of [DOATAP](https://www.doatap.gr) ([Medical MCQA](https://huggingface.co/datasets/ilsp/medical_mcqa_greek)).
 
-Our evaluation for Llama-Krikri-8B is performed in a few-shot setting, consistent with the settings in the [Open LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). We can see that our training enhances performance across all Greek test sets by a **+10.8%** average improvement. The results for the Greek test sets are shown in the following table:
+We can see that our training enhances performance across all Greek test sets by a **+10.8%** average improvement. The results for the Greek test sets are shown in the following table:
 
 |                | Medical MCQA EL (15-shot) | Belebele EL (5-shot) | HellaSwag EL (10-shot) | ARC-Challenge EL (25-shot) | TruthfulQA MC2 EL (0-shot) | MMLU EL (5-shot) | Average |
 |----------------|----------------|-------------|--------------|------------------|-------------------|---------|---------|
@@ -107,6 +114,8 @@ Our evaluation for Llama-Krikri-8B is performed in a few-shot setting, consisten
 
 ## English Benchmarks
 
+We can also see that our training methodology not only mitigates catastrophic forgetting effectively, but also improves average performance across all English test sets by **+0.8%**. The results for the English test sets are shown in the following table:
+
 |                | Winogrande (5-shot) | Belebele (5-shot) | HellaSwag (10-shot) | ARC-Challenge (25-shot) | TruthfulQA MC2 (0-shot) | MMLU (5-shot) | Average |
 |----------------|----------------|-------------|--------------|------------------|-------------------|---------|---------|
 | Meltemi 7B v1.5 | 73.4%         | 77.7%       | 79.6%        | 54.1%            | 40.5%             | 56.9%   | 63.7%   |