Evaluating Audio Reasoning with Big Bench Audio
ā¢
13
George Cameron
georgewritescode
AI & ML interests
None yet
Recent Activity
liked
a dataset
5 days ago
ArtificialAnalysis/big_bench_audio
updated
a dataset
5 days ago
ArtificialAnalysis/hf-assets
updated
a Space
about 1 month ago
ArtificialAnalysis/Video-Generation-Arena-Leaderboard
Articles
Organizations
Posts
2
Post
1017
Visualization of GPT-4o breaking away from the quality & speed trade-off curve the LLMs have followed thus far āļø
Key GPT-4o takeaways
ā£ GPT-4o not only offers the highest quality, it also sits amongst the fastest LLMs
ā£ For those with speed/latency-sensitive use cases, where previously Claude 3 Haiku or Mixtral 8x7b were leaders, GPT-4o is now a compelling option (though significantly more expensive)
ā£ Previously Groq was the only provider to break from the curve using its own LPU chips. OpenAI has done it on Nvidia hardware (one can imagine the potential for GPT-4o on Groq)
š How did they do it? Will follow up with more analysis on this but potential approaches include a very large but sparse MoE model (similar to Snowflake's Arctic) and improvements in data quality (likely to have driven much of Llama 3's impressive quality relative to parameter count)
Notes: Throughput represents the median across providers over the last 14 days of measurements (8x per day)
Data is present on our HF leaderboard: ArtificialAnalysis/LLM-Performance-Leaderboard and graphs present on our website
Key GPT-4o takeaways
ā£ GPT-4o not only offers the highest quality, it also sits amongst the fastest LLMs
ā£ For those with speed/latency-sensitive use cases, where previously Claude 3 Haiku or Mixtral 8x7b were leaders, GPT-4o is now a compelling option (though significantly more expensive)
ā£ Previously Groq was the only provider to break from the curve using its own LPU chips. OpenAI has done it on Nvidia hardware (one can imagine the potential for GPT-4o on Groq)
š How did they do it? Will follow up with more analysis on this but potential approaches include a very large but sparse MoE model (similar to Snowflake's Arctic) and improvements in data quality (likely to have driven much of Llama 3's impressive quality relative to parameter count)
Notes: Throughput represents the median across providers over the last 14 days of measurements (8x per day)
Data is present on our HF leaderboard: ArtificialAnalysis/LLM-Performance-Leaderboard and graphs present on our website
Post
2324
Excited to bring our benchmarking leaderboard of >100 LLM API endpoints to HF!
Speed and price are often just as important as quality when building applications with LLMs. We bring together all the data you need to consider all three when you need to pick a model and API provider.
Coverage:
ā£ Quality (Index of evals, MMLU, Chatbot Arena, HumanEval, MT-Bench)
ā£ Throughput (tokens/s: median, P5, P25, P75, P95)
ā£ Latency (TTFT: median, P5, P25, P75, P95)
ā£ Context window
ā£ OpenAI library compatibility
Link to Space: ArtificialAnalysis/LLM-Performance-Leaderboard
Blog post: https://huggingface.co/blog/leaderboard-artificial-analysis
Speed and price are often just as important as quality when building applications with LLMs. We bring together all the data you need to consider all three when you need to pick a model and API provider.
Coverage:
ā£ Quality (Index of evals, MMLU, Chatbot Arena, HumanEval, MT-Bench)
ā£ Throughput (tokens/s: median, P5, P25, P75, P95)
ā£ Latency (TTFT: median, P5, P25, P75, P95)
ā£ Context window
ā£ OpenAI library compatibility
Link to Space: ArtificialAnalysis/LLM-Performance-Leaderboard
Blog post: https://huggingface.co/blog/leaderboard-artificial-analysis
models
None public yet
datasets
None public yet