microsoft/Phi-3.5-MoE-instruct
Text Generation
โข
Updated
โข
45.5k
โข
540
Jฤdrzej Grabala
jgitsolutions
AI & ML interests
Local Drive Human Overseered System of Agents, LLMs, Langchains & other useful stuff on mid-to-low-end of commercial hardware.
Recent Activity
liked
a model
5 days ago
microsoft/Phi-3.5-MoE-instruct
liked
a Space
5 days ago
Lightricks/LTX-Video-Playground
liked
a Space
21 days ago
cutechicken/tankwar
Organizations
jgitsolutions's activity
liked
a
model
5 days ago
liked
a
Space
5 days ago
liked
2
Spaces
21 days ago
liked
2
Spaces
28 days ago
liked
a
Space
29 days ago
reacted to
anakin87's
post with ๐
about 1 month ago
Post
1104
Ok, you're finally convinced that synthetic data works... โ๏ธ
๐๐จ๐ฐ ๐ฒ๐จ๐ฎ ๐ฐ๐๐ง๐ญ ๐ญ๐จ ๐ ๐๐ง๐๐ซ๐๐ญ๐ ๐๐ง ๐ข๐ง๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ข๐จ๐ง ๐๐๐ญ๐๐ฌ๐๐ญ ๐๐จ๐ซ ๐๐ข๐ง๐-๐ญ๐ฎ๐ง๐ข๐ง๐ ๐ข๐ง ๐ ๐ฅ๐๐ง๐ ๐ฎ๐๐ ๐ ๐จ๐ญ๐ก๐๐ซ ๐ญ๐ก๐๐ง ๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก.
But how do you get started?
I explore how to do this with Magpie in my new article
https://huggingface.co/blog/anakin87/multilingual-magpie
---
๐ฆโโฌ ๐๐ก๐๐ญ ๐ข๐ฌ ๐๐๐ ๐ฉ๐ข๐?
It's a recent technique for creating synthetic instruction datasets.
Magpie is based on a simple but ingenious idea ๐
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction
Here's an example:
model: Llama-3-8B-Instruct
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
generated user instruction: "What are some of the responsibilities of a commercial pilot?"
You can then feed this instruction back into the same model to get the assistant response.
By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.
๐ช The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.
๐ง๐๐๐ง๐๐ซ๐๐ญ๐ข๐ง๐ ๐ง๐จ๐ง-๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก ๐๐๐ญ๐
Most Language Models are primarily trained on English texts, so they tend to produce data in English.
How can we overcome this?
Earlier approaches were complex or costly.
Then @mrm8488 found a simple solution: add the target language to the pre-query template.
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".
This method works for Spanish and German!
โ Unfortunately, it does not work well for other languages (๐ฎ๐น, ๐ณ๐ฑ, ...)
๐
๐๐จ๐ฐ ๐ฒ๐จ๐ฎ ๐ฐ๐๐ง๐ญ ๐ญ๐จ ๐ ๐๐ง๐๐ซ๐๐ญ๐ ๐๐ง ๐ข๐ง๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ข๐จ๐ง ๐๐๐ญ๐๐ฌ๐๐ญ ๐๐จ๐ซ ๐๐ข๐ง๐-๐ญ๐ฎ๐ง๐ข๐ง๐ ๐ข๐ง ๐ ๐ฅ๐๐ง๐ ๐ฎ๐๐ ๐ ๐จ๐ญ๐ก๐๐ซ ๐ญ๐ก๐๐ง ๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก.
But how do you get started?
I explore how to do this with Magpie in my new article
https://huggingface.co/blog/anakin87/multilingual-magpie
---
๐ฆโโฌ ๐๐ก๐๐ญ ๐ข๐ฌ ๐๐๐ ๐ฉ๐ข๐?
It's a recent technique for creating synthetic instruction datasets.
Magpie is based on a simple but ingenious idea ๐
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction
Here's an example:
model: Llama-3-8B-Instruct
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
generated user instruction: "What are some of the responsibilities of a commercial pilot?"
You can then feed this instruction back into the same model to get the assistant response.
By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.
๐ช The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.
๐ง๐๐๐ง๐๐ซ๐๐ญ๐ข๐ง๐ ๐ง๐จ๐ง-๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก ๐๐๐ญ๐
Most Language Models are primarily trained on English texts, so they tend to produce data in English.
How can we overcome this?
Earlier approaches were complex or costly.
Then @mrm8488 found a simple solution: add the target language to the pre-query template.
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".
This method works for Spanish and German!
โ Unfortunately, it does not work well for other languages (๐ฎ๐น, ๐ณ๐ฑ, ...)
๐
- 1 reply
reacted to
yongchanghao's
post with ๐๐ฅ
about 1 month ago
Post
3747
We just released a paper (NeuZip) that compresses VRAM in a lossless manner to run larger models. This should be particularly useful when VRAM is insufficient during training/inference. Specifically, we look inside each floating number and find that the exponents are highly compressible (as shown in the figure below).
Read more about the work at NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks (2410.20650)
Read more about the work at NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks (2410.20650)
reacted to
PLB's
post with ๐
about 1 month ago
Post
1863
โ ๏ธ People selling AI chatbots for websites hate us.
Add an open source chat assistant on your website in 5 minutes: https://github.com/phospho-app/ai-chat-bubble
How does it work ?
- You give an URL
- The AI assistant crawls the website content and embed it
- Add it to your frontend in one line of code
- People on your website can ask the assistant questions
Powered by BAAI/bge-small-en-v1.5 and Mistral AI
Add an open source chat assistant on your website in 5 minutes: https://github.com/phospho-app/ai-chat-bubble
How does it work ?
- You give an URL
- The AI assistant crawls the website content and embed it
- Add it to your frontend in one line of code
- People on your website can ask the assistant questions
Powered by BAAI/bge-small-en-v1.5 and Mistral AI
- 5 replies
reacted to
LukeNeumann's
post with ๐
about 1 month ago
Post
1859
Hello Hugging Face community!
I wanted to introduce myself and my company @Overlaiapp . We are a collective of filmmakers, photographers, and AI engineers working on high resolution (8K+) training data.
We plan to share a lot of our datasets with the community and are kicking things off with two curated datasets:
- Overlaiai/OregonCoastin4K
- Overlaiai/SubArcticPolarBear
Overlai.ai Dataset Features
๐ฅ Oversampled: Every clip is captured in stunning 8K resolution, delivering rich detail ideal for fine tuning scenic landscapes and ocean dynamics.
๐ธ Variance: Includes close-up details, slow-motion footage of crashing waves, sweeping landscapes, and wildlife shots.
๐ Detailed Metadata: Every clip is paired with structured metadata, including creative descriptions, precise camera movements, lens information, field of view calculations, and shot settings, ensuring AI models can fully understand and replicate real-world cinematography with accuracy.
โ๏ธ Consistency: Re-thinking training data at the point of capture by "overshooting" a subject, enabling models to learn more nuanced relationships and views across scenes.
๐ Light: Shot during early morning and sunset light for optimal color contrast and dynamic range, maximizing visual quality for color and lighting-sensitive tasks.
๐ Curation: Curated specifically for machine learning, providing clean, high-quality data for next generation model training.
I wanted to introduce myself and my company @Overlaiapp . We are a collective of filmmakers, photographers, and AI engineers working on high resolution (8K+) training data.
We plan to share a lot of our datasets with the community and are kicking things off with two curated datasets:
- Overlaiai/OregonCoastin4K
- Overlaiai/SubArcticPolarBear
Overlai.ai Dataset Features
๐ฅ Oversampled: Every clip is captured in stunning 8K resolution, delivering rich detail ideal for fine tuning scenic landscapes and ocean dynamics.
๐ธ Variance: Includes close-up details, slow-motion footage of crashing waves, sweeping landscapes, and wildlife shots.
๐ Detailed Metadata: Every clip is paired with structured metadata, including creative descriptions, precise camera movements, lens information, field of view calculations, and shot settings, ensuring AI models can fully understand and replicate real-world cinematography with accuracy.
โ๏ธ Consistency: Re-thinking training data at the point of capture by "overshooting" a subject, enabling models to learn more nuanced relationships and views across scenes.
๐ Light: Shot during early morning and sunset light for optimal color contrast and dynamic range, maximizing visual quality for color and lighting-sensitive tasks.
๐ Curation: Curated specifically for machine learning, providing clean, high-quality data for next generation model training.
reacted to
BlinkDL's
post with ๐ฅ
about 1 month ago
Post
3959
RWKV-6-world-v3 (+3.1T tokens) is our best multilingual 7B model as of now:
BlinkDL/rwkv-6-world
It's 100% RNN and attention-free. MMLU 54.2% (previous world-v2.1 = 47.9%. note: without eval-boosting tricks such as annealing).
RWKV-7-world-v4 soon :)
It's 100% RNN and attention-free. MMLU 54.2% (previous world-v2.1 = 47.9%. note: without eval-boosting tricks such as annealing).
RWKV-7-world-v4 soon :)
liked
5
Spaces
about 1 month ago
liked
2
models
about 2 months ago
