2 OLMo 2 Furious
Paper
•
2501.00656
•
Published
•
15
alkinun
AtAndDev
AI & ML interests
LLMs, Alignment, Merging, Unsloth, DPO, SFT, ORPO, SPIN..
Recent Activity
reacted
to
davanstrien's
post
with 🤝
about 15 hours ago
The https://huggingface.co/datasets/data-is-better-together/fineweb-c dataset is growing!
This week a few more languages have got 1,000 annotations for the educational quality of data from https://huggingface.co/datasets/HuggingFaceFW/fineweb-2.
Why should you care?
The quality of pre-training data can have a big impact on the performance of downstream language models trained on that data (https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1).
Being able to filter by educational quality is on way of improving the quality of the data you use for training an LLM. Very importantly this approach can also reduce the amount of data needed for pertaining.
Why not use an LLM?
LLMs can be used to annotate educational quality for a subset of data. This data can then be used to train a smaller encoder only model to label the full dataset. However, this may not work well for languages outside of english. This is where fineweb-c (community) comes in.
The community is annotating the educational quality of fineweb2 data. Currently 114 languages have some annotations. These annotations will enable a number of things:
- Evaluate whether an LLM can label the educational quality for texts in that language well
- Directly be used for training quality classifiers
- Help discover other rules and huerisitcs for refining fineweb2 further for different languages.
This week the following languages where done:
Swedish thanks to: @Lauler @AntonVic @ohallstrom @bjarlestam @menbom @Ekgren @apsod
Ukrainian thanks to: @hannayukhymenko @robinhad @realPivo @RabotiahovDmytro @reciprocate
Assamese thanks to: @moyoor97 @Arpanjyoti @nawaf-helmi123 @pahigogoi1 @aelhence @kishorekashyap
Want to learn more: https://huggingface.co/blog/davanstrien/fineweb2-community
Contribute yourself here: https://huggingface.co/spaces/data-is-better-together/fineweb-c
reacted
to
davanstrien's
post
with 🔥
about 15 hours ago
The https://huggingface.co/datasets/data-is-better-together/fineweb-c dataset is growing!
This week a few more languages have got 1,000 annotations for the educational quality of data from https://huggingface.co/datasets/HuggingFaceFW/fineweb-2.
Why should you care?
The quality of pre-training data can have a big impact on the performance of downstream language models trained on that data (https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1).
Being able to filter by educational quality is on way of improving the quality of the data you use for training an LLM. Very importantly this approach can also reduce the amount of data needed for pertaining.
Why not use an LLM?
LLMs can be used to annotate educational quality for a subset of data. This data can then be used to train a smaller encoder only model to label the full dataset. However, this may not work well for languages outside of english. This is where fineweb-c (community) comes in.
The community is annotating the educational quality of fineweb2 data. Currently 114 languages have some annotations. These annotations will enable a number of things:
- Evaluate whether an LLM can label the educational quality for texts in that language well
- Directly be used for training quality classifiers
- Help discover other rules and huerisitcs for refining fineweb2 further for different languages.
This week the following languages where done:
Swedish thanks to: @Lauler @AntonVic @ohallstrom @bjarlestam @menbom @Ekgren @apsod
Ukrainian thanks to: @hannayukhymenko @robinhad @realPivo @RabotiahovDmytro @reciprocate
Assamese thanks to: @moyoor97 @Arpanjyoti @nawaf-helmi123 @pahigogoi1 @aelhence @kishorekashyap
Want to learn more: https://huggingface.co/blog/davanstrien/fineweb2-community
Contribute yourself here: https://huggingface.co/spaces/data-is-better-together/fineweb-c
Organizations
AtAndDev's activity
upvoted
a
paper
8 days ago
upvoted
a
paper
11 days ago
upvoted
a
paper
23 days ago
upvoted
a
paper
28 days ago
upvoted
a
paper
29 days ago
upvoted
a
paper
about 1 month ago
upvoted
a
collection
about 1 month ago
upvoted
2
papers
about 1 month ago
upvoted
2
collections
about 2 months ago
upvoted
4
papers
about 2 months ago
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
Paper
•
2403.09629
•
Published
•
75
V-STaR: Training Verifiers for Self-Taught Reasoners
Paper
•
2402.06457
•
Published
•
9
Let's Verify Step by Step
Paper
•
2305.20050
•
Published
•
10
Learn Beyond The Answer: Training Language Models with Reflection for Mathematical Reasoning
Paper
•
2406.12050
•
Published
•
19
upvoted
a
collection
about 2 months ago
upvoted
4
papers
2 months ago
Large Language Models Orchestrating Structured Reasoning Achieve Kaggle Grandmaster Level
Paper
•
2411.03562
•
Published
•
65
What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective
Paper
•
2410.23743
•
Published
•
59
LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations
Paper
•
2410.02707
•
Published
•
48
GPT-4o System Card
Paper
•
2410.21276
•
Published
•
83