main merged to 1.1

README.md

@@ -87,7 +87,7 @@ The pre-training corpus contains text in 35 European languages and code.
 
 ### Hyperparameters
 
-The full list of hyperparameters for each model can be found [here](https://github.com/langtech-bsc/salamandra/blob/main/configs/bsc_7b.yaml).
+The full list of hyperparameters for each model can be found [here](https://github.com/langtech-bsc/salamandra/tree/main/configs).
 
 ### Architecture
 
@@ -141,7 +141,7 @@ All models were trained on [MareNostrum 5](https://www.bsc.es/ca/marenostrum/mar
 operated by Barcelona Supercomputing Center.
 
 The accelerated partition is composed of 1,120 nodes with the following specifications:
-- 4x Nvidia Hopper GPUs with 64GB HBM2 memory
+- 4x Nvidia Hopper GPUs with 64 HBM2 memory
 - 2x Intel Sapphire Rapids 8460Y+ at 2.3Ghz and 32c each (64 cores)
 - 4x NDR200 (BW per node 800Gb/s)
 - 512 GB of Main memory (DDR5)
@@ -309,8 +309,8 @@ This adjustment resulted in a total of 2.68 trillion tokens, distributed as outl
 
 ![lang distrib](./images/corpus_languages_1.1.png)
 
-The pretraining corpus is predominantly composed of data from Colossal OSCAR, which contributes a significant 53,05% of the total tokens. 
-Following this, Starcoder provides 13,67%, and FineWeb-Edu (350BT subset) adds 10,24%. The next largest sources are HPLT at 4,21% and French-PD at 3,59%. 
+The pretraining corpus is predominantly composed of data from Colossal OSCAR, which contributes a significant 53.05% of the total tokens. 
+Following this, Starcoder provides 13.67%, and FineWeb-Edu (350BT subset) adds 10.24%. The next largest sources are HPLT at 4.21% and French-PD at 3.59%. 
 Other notable contributions include MaCoCu, Legal-ES, and EurLex, each contributing around 1.72% to 1.41%. 
 These major sources collectively form the bulk of the corpus, ensuring a rich and diverse dataset for training the language model. 
 The remaining 10% comes from smaller sources in various languages.
@@ -590,8 +590,8 @@ especially if the content originates from less-regulated sources or user-generat
 **How was the data collected?**
 
 This dataset is constituted by combining several sources, whose acquisition methods can be classified into three groups:
-- Web-sourced datasets with some preprocessing available under permissive license.
-- Domain-specific or language-specific raw crawls, always respecting robots.txt.
+- Web-sourced datasets with some preprocessing available under permissive license (p.e. Common Crawl).
+- Domain-specific or language-specific raw crawls, always respecting robots.txt (p.e. Spanish Crawling).
 - Manually curated data obtained through collaborators, data providers (by means of legal assignment agreements) or open source projects
   (p.e. CATalog).
 
@@ -644,7 +644,7 @@ The original raw data was not kept.
 
 **Is the software that was used to preprocess/clean/label the data available? If so, please provide a link or other access point.**
 
-Yes, the preprocessing and filtering software is open-sourced. The [CURATE](https://github.com/langtech-bsc/CURATE) pipeline was used for CATalog and other curated sources, 
+Yes, the preprocessing and filtering software is open-sourced. The [CURATE](https://github.com/langtech-bsc/CURATE) pipeline was used for Spanish Crawling and CATalog, 
 and the [Ungoliant](https://github.com/oscar-project/ungoliant) pipeline was used for the OSCAR project.
 
 #### Uses
@@ -724,7 +724,7 @@ We only use tasks that are either human generated, human translated, or with a s
 
 During the implementation of the evaluation we observed a series of issues worth considering when replicating and interpreting the results presented. These issues include ≈1.5% variances in performance in some tasks depending on the version of the `transformers` library used, and depending on the use (or lack of use) of tensor parallelism when loading a model. When implementing existing tasks, we carry out a comprehensive quality evaluation of the dataset, the Harness task itself, and what kind of input models see during evaluation. Our implementation (see links above) addresses multiple existing problems such as errors in datasets and prompts, and lack of pre-processing. All this means that results will vary if using other Harness implementations, and may slightly vary depending on the replication setup.
 
-It should be noted that these results are subject to all the drawbacks of every current gold-standard evaluation, and that the figures do not fully represent the model's capabilities and potential. We thus advise caution when reading and interpreting the results.
+It should be noted that these results are subject to all the drawbacks of every current gold-standard evaluation, and that the figures do not fully represent the models capabilities and potential. We thus advise caution when reading and interpreting the results.
 
 A full list of results compared to other baselines, a discussion of the model's performance across tasks and its implications, and details regarding problem-solving with task implementation will soon be available in the technical report.
 
@@ -1065,7 +1065,7 @@ We report that while performance is high (accuracies between 0.69 and 0.87 depen
 the model performs very poorly in ambiguous settings, which is indicative of the presence of societal biases which need to be addressed in post-training phases.
 
 We additionally analyse model generations using the Regard dataset and classifier in Catalan, Spanish, and English using backtranslation and manual revision of the 
-translations. We find no statistically significant difference in regard between majority and minority groups for any regard type,
+translations. We find no statistically significant difference in regard between majority and minority groups for any regard types,
 with the exception of negative regard in Catalan where model generations are actually slightly worse for social majorities. 
 Our analyses on societal biases show that while these biases are capable of interfering with model performance as expressed in the results on the BBQ dataset,
 their tendency for representational harm is limited given the results of the Regard dataset. We highlight that our analyses of these biases are by no means exhaustive
@@ -1075,7 +1075,7 @@ in future work.
 Our cognitive bias analysis focuses on positional effects in 0-shot settings, and majority class bias in few-shot settings.
 For positional effects, we leverage the ARC Multiple Choice Question dataset (Clark et al., 2018).
 We observe moderate to strong primacy effects, whereby the model shows a preference for answers towards the beginning of the list of provided answers.
-We measure the effects of majority class effects in few-shot settings using SST-2 (Socher et al., 2013). We detect moderate effects,
+We measure effects of majority class effects in few-shot settings using SST-2 (Socher et al., 2013). We detect moderate effects,
 implying that outputs can be influenced by the prompts.
 
 We highlight that these results can be expected from a pretrained model that has not yet been instruction-tuned or aligned.
@@ -1133,4 +1133,4 @@ Technical report coming soon.
 |:---:|:---:|:---:|
 |2B| [Link](https://huggingface.co/BSC-LT/salamandra-2b) | [Link](https://huggingface.co/BSC-LT/salamandra-2b-instruct) |
 |7B| [Link](https://huggingface.co/BSC-LT/salamandra-7b) | [Link](https://huggingface.co/BSC-LT/salamandra-7b-instruct) |
-|40B| [Link](https://huggingface.co/BSC-LT/ALIA-40b) | WiP |
+|40B| [Link](https://huggingface.co/BSC-LT/ALIA-40b) | WiP |