deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
Text Generation
•
Updated
•
1.43M
•
•
1.23k
Bulat
Bulat15g
AI & ML interests
None yet
Recent Activity
liked
a model
about 1 month ago
deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
liked
a dataset
4 months ago
upstage/dp-bench
upvoted
an
article
5 months ago
Llama can now see and run on your device - welcome Llama 3.2
Organizations
Bulat15g's activity
liked
a
model
about 1 month ago
liked
a
dataset
4 months ago
upvoted
an
article
5 months ago
Article
Llama can now see and run on your device - welcome Llama 3.2
•
183
liked
a
dataset
7 months ago
upvoted
a
collection
8 months ago
liked
a
model
8 months ago
upvoted
an
article
9 months ago
Article
Uncensor any LLM with abliteration
By
•
•
452
liked
a
dataset
10 months ago
liked
a
model
10 months ago
upvoted
2
collections
10 months ago
upvoted
a
collection
12 months ago
reacted to
akhaliq's
post with 👍
about 1 year ago
Post
MobileLLM
Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (2402.14905)
paper addresses the growing need for efficient large language models (LLMs) on mobile devices, driven by increasing cloud costs and latency concerns. We focus on designing top-quality LLMs with fewer than a billion parameters, a practical choice for mobile deployment. Contrary to prevailing belief emphasizing the pivotal role of data and parameter quantity in determining model quality, our investigation underscores the significance of model architecture for sub-billion scale LLMs. Leveraging deep and thin architectures, coupled with embedding sharing and grouped-query attention mechanisms, we establish a strong baseline network denoted as MobileLLM, which attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M state-of-the-art models. Additionally, we propose an immediate block-wise weight sharing approach with no increase in model size and only marginal latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover, MobileLLM model family shows significant improvements compared to previous sub-billion models on chat benchmarks, and demonstrates close correctness to LLaMA-v2 7B in API calling tasks, highlighting the capability of small models for common on-device use cases.
Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (2402.14905)
paper addresses the growing need for efficient large language models (LLMs) on mobile devices, driven by increasing cloud costs and latency concerns. We focus on designing top-quality LLMs with fewer than a billion parameters, a practical choice for mobile deployment. Contrary to prevailing belief emphasizing the pivotal role of data and parameter quantity in determining model quality, our investigation underscores the significance of model architecture for sub-billion scale LLMs. Leveraging deep and thin architectures, coupled with embedding sharing and grouped-query attention mechanisms, we establish a strong baseline network denoted as MobileLLM, which attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M state-of-the-art models. Additionally, we propose an immediate block-wise weight sharing approach with no increase in model size and only marginal latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover, MobileLLM model family shows significant improvements compared to previous sub-billion models on chat benchmarks, and demonstrates close correctness to LLaMA-v2 7B in API calling tasks, highlighting the capability of small models for common on-device use cases.
liked
a
model
over 1 year ago
upvoted
a
paper
over 1 year ago
liked
a
model
almost 2 years ago
liked
a
Space
almost 2 years ago
liked
a
model
about 2 years ago
liked
a
Space
about 2 years ago
liked
a
model
almost 3 years ago
