Meta Llama

Enterprise

company

Verified

https://ai.meta.com/llama/

meta-llama

Activity Feed

AI & ML interests

None defined yet.

Recent Activity

wukaixingxp new activity 6 days ago

meta-llama/Llama-3.2-11B-Vision-Instruct:Unexpected Output using Official Example Code

Hamid-Nazeri new activity 22 days ago

meta-llama/Llama-3.3-70B-Instruct:Model card updates

vontimitta new activity 26 days ago

meta-llama/Llama-3.3-70B-Instruct:Base Model?

View all activity

meta-llama's activity

shellysheynin

authored a paper 5 days ago

Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation

Paper • 2501.03059 • Published 6 days ago • 18

wukaixingxp

in meta-llama/Llama-3.2-11B-Vision-Instruct 6 days ago

Unexpected Output using Official Example Code

#71 opened 2 months ago by

jenniferL

Hamid-Nazeri

in meta-llama/Llama-3.3-70B-Instruct 22 days ago

Model card updates

#28 opened about 1 month ago by

pcuenq

vontimitta

in meta-llama/Llama-3.3-70B-Instruct 26 days ago

Base Model?

#32 opened 30 days ago by

User8213

Is EU use permitted with THIS model?

#35 opened 27 days ago by

ptrdvn

vontimitta

in meta-llama/Meta-Llama-3-70B-Instruct 28 days ago

Update to latest version of the Llama 3.3 70B Instruct model

#71 opened about 1 month ago by

reach-vb

vontimitta

in meta-llama/Llama-3.1-70B-Instruct 28 days ago

Add new model version to the model card

#42 opened about 1 month ago by

reach-vb

Narsil

posted an update about 1 month ago

Post

1092

Performance leap: TGI v3 is out. Processes 3x more tokens, 13x faster than vLLM on long prompts. Zero config !

3x more tokens.

By reducing our memory footprint, we’re able to ingest many more tokens and more dynamically than before. A single L4 (24GB) can handle 30k tokens on llama 3.1-8B, while vLLM gets barely 10k. A lot of work went into reducing the footprint of the runtime and its effect are best seen on smaller constrained environments.
13x faster

On long prompts (200k+ tokens) conversation replies take 27.5s in vLLM, while it takes only 2s in TGI. How so ? We keep the initial conversation around, so when a new reply comes in, we can answer almost instantly. The overhead of the lookup is ~5us. Thanks @Dani ël de Kok for the beast data structure.
Zero config

That’s it. Remove all the flags your are using and you’re likely to get the best performance. By evaluating the hardware and model, TGI carefully selects automatic values to give best performance. In production, we don’t have any flags anymore in our deployments. We kept all existing flags around, they may come in handy in niche scenarios.

Read more: https://huggingface.co/docs/text-generation-inference/conceptual/chunking