SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
Paper
•
2409.07440
•
Published
•
6
Open CoT Leaderboard
community
AI & ML interests
Chain of Thought, LLM Evaluation
Recent Activity
View all activity
cot-leaderboard's activity
yakazimir
authored
a
paper
about 2 months ago
ggbetz
updated
a
Space
about 2 months ago
ggbetz
updated
a
dataset
about 2 months ago
Update leaderboard for model HuggingFaceTB/SmolLM2-1.7B-Instruct
1
#131 opened about 2 months ago
by
ggbetz
Upload results for model HuggingFaceTB/SmolLM2-1.7B-Instruct
1
#1023 opened about 2 months ago
by
ggbetz
Upload results for model HuggingFaceTB/SmolLM2-1.7B-Instruct
1
#1024 opened about 2 months ago
by
ggbetz
Upload results for model HuggingFaceTB/SmolLM2-1.7B-Instruct
1
#1025 opened about 2 months ago
by
ggbetz
Upload results for model HuggingFaceTB/SmolLM2-1.7B-Instruct
1
#1026 opened about 2 months ago
by
ggbetz
Upload results for model HuggingFaceTB/SmolLM2-1.7B-Instruct
1
#1027 opened about 2 months ago
by
ggbetz
Upload results for model HuggingFaceTB/SmolLM2-1.7B-Instruct
1
#1028 opened about 2 months ago
by
ggbetz
Upload results for model HuggingFaceTB/SmolLM2-1.7B-Instruct
1
#1029 opened about 2 months ago
by
ggbetz
Upload results for model HuggingFaceTB/SmolLM2-1.7B-Instruct
1
#1030 opened about 2 months ago
by
ggbetz
ggbetz
updated
3
datasets
about 2 months ago
Post
1456
Hi, just a brief follow-up on our Guided Reasoning (GuiR) system:
I've created a template space that facilitates testing:
1. Duplicate space logikon/guir-chat
2. Setup your own inference servers and provide details in config file
3. Add api keys as secrets
4. Your personal GuiR playground is ready
Cheers, Gregor
I've created a template space that facilitates testing:
1. Duplicate space logikon/guir-chat
2. Setup your own inference servers and provide details in config file
3. Add api keys as secrets
4. Your personal GuiR playground is ready
Cheers, Gregor
Post
1179
🧭 Guided Reasoning
👋Hi everyone,
We've been releasing Guided Reasoning:
Our AI guides walk your favorite LLM through complex reasoning problems.
🎯 Goals:
1️⃣ Reliability. AIs consistently follow reasoning methods.
2️⃣ Self-explainability. AIs see reasoning protocols and can explain internal deliberation.
3️⃣ Contestability. Users may amend AI reasoning and revise plausibility assessments.
Try out Guided Reasoning with our light demo chatbot, powered by 🤗 HuggingFace's free Inference Api and small LLMs. (Sorry for poor latency and limited availability -- we are currently searching for 💸 compute sponsors to run more powerful models, faster, and optimize guided reasoning performance.)
Built on top of Logikon's open-source AI reasoning analytics.
Demo chat app: logikon/benjamin-chat
Github: https://github.com/logikon-ai/logikon
Technical report: https://arxiv.org/abs/2408.16331
➡️ Check it out and get involved! Looking forward to hearing from you.
👋Hi everyone,
We've been releasing Guided Reasoning:
Our AI guides walk your favorite LLM through complex reasoning problems.
🎯 Goals:
1️⃣ Reliability. AIs consistently follow reasoning methods.
2️⃣ Self-explainability. AIs see reasoning protocols and can explain internal deliberation.
3️⃣ Contestability. Users may amend AI reasoning and revise plausibility assessments.
Try out Guided Reasoning with our light demo chatbot, powered by 🤗 HuggingFace's free Inference Api and small LLMs. (Sorry for poor latency and limited availability -- we are currently searching for 💸 compute sponsors to run more powerful models, faster, and optimize guided reasoning performance.)
Built on top of Logikon's open-source AI reasoning analytics.
Demo chat app: logikon/benjamin-chat
Github: https://github.com/logikon-ai/logikon
Technical report: https://arxiv.org/abs/2408.16331
➡️ Check it out and get involved! Looking forward to hearing from you.
Post
1440
🥇Open CoT Leaderboard
We're delighted to announce the [Open CoT Leaderboard]( logikon/open_cot_leaderboard) on 🤗 Spaces.
Unlike other LLM performance leaderboards, the Open CoT Leaderboard is not tracking absolute benchmark accuracies, but relative **accuracy gains** due to **chain-of-thought**.
Eval datasets that underpin the leaderboard are hosted [here](https://huggingface.co/cot-leaderboard).
Feedback and suggestions more than welcome.
@clefourrier
We're delighted to announce the [Open CoT Leaderboard]( logikon/open_cot_leaderboard) on 🤗 Spaces.
Unlike other LLM performance leaderboards, the Open CoT Leaderboard is not tracking absolute benchmark accuracies, but relative **accuracy gains** due to **chain-of-thought**.
Eval datasets that underpin the leaderboard are hosted [here](https://huggingface.co/cot-leaderboard).
Feedback and suggestions more than welcome.
@clefourrier
-
- 5 replies
yakazimir
authored
2
papers
11 months ago