File size: 8,431 Bytes
c376079 c44d5b7 c376079 c44d5b7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 |
---
license: llama3
model_name: Llama-3-Groq-70B-Tool-Use
base_model: Groq/Llama-3-Groq-70B-Tool-Use
model_creator: Groq
inference: false
pipeline_tag: text-generation
quantized_by: Second State Inc.
language:
- en
---
<!-- header start -->
<!-- 200823 -->
<div style="width: auto; margin-left: auto; margin-right: auto">
<img src="https://github.com/LlamaEdge/LlamaEdge/raw/dev/assets/logo.svg" style="width: 100%; min-width: 400px; display: block; margin: auto;">
</div>
<hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
<!-- header end -->
# Llama-3-Groq-70B-Tool-Use-GGUF
## Original Model
[Groq/Llama-3-Groq-70B-Tool-Use](https://huggingface.co/Groq/Llama-3-Groq-70B-Tool-Use)
## Run with LlamaEdge
- LlamaEdge version: [v0.12.5](https://github.com/LlamaEdge/LlamaEdge/releases/tag/0.12.5)
- Prompt template
- Prompt type: `groq-llama3-tool`
- Prompt string
```text
<|start_header_id|>system<|end_header_id|>
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
<tool_call>
{"name": <function-name>,"arguments": <args-dict>}
</tool_call>
Here are the available tools:
<tools> {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"description": "The temperature unit to use. Infer this from the users location.",
"enum": [
"celsius",
"fahrenheit"
]
}
},
"required": [
"location",
"unit"
]
}
}
{
"name": "predict_weather",
"description": "Predict the weather in 24 hours",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"description": "The temperature unit to use. Infer this from the users location.",
"enum": [
"celsius",
"fahrenheit"
]
}
},
"required": [
"location",
"unit"
]
}
} </tools><|eot_id|><|start_header_id|>user<|end_header_id|>
What is the weather like in San Francisco in Celsius?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```
- Context size: `8192`
- Run as LlamaEdge service
```bash
wasmedge --dir .:. --nn-preload default:GGML:AUTO:Llama-3-Groq-70B-Tool-Use-Q5_K_M.gguf \
llama-api-server.wasm \
--prompt-template groq-llama3-tool \
--ctx-size 8192 \
--model-name Llama-3-Groq-70B
```
- Run as LlamaEdge command app
```bash
wasmedge --dir .:. --nn-preload default:GGML:AUTO:Llama-3-Groq-70B-Tool-Use-Q5_K_M.gguf \
llama-chat.wasm \
--prompt-template groq-llama3-tool \
--ctx-size 8192
```
## Quantized GGUF Models
| Name | Quant method | Bits | Size | Use case |
| ---- | ---- | ---- | ---- | ----- |
| [Llama-3-Groq-70B-Tool-Use-Q2_K.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q2_K.gguf) | Q2_K | 2 | 26.4 GB| smallest, significant quality loss - not recommended for most purposes |
| [Llama-3-Groq-70B-Tool-Use-Q3_K_L.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q3_K_L.gguf) | Q3_K_L | 3 | 37.1 GB| small, substantial quality loss |
| [Llama-3-Groq-70B-Tool-Use-Q3_K_M.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q3_K_M.gguf) | Q3_K_M | 3 | 34.3 GB| very small, high quality loss |
| [Llama-3-Groq-70B-Tool-Use-Q3_K_S.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q3_K_S.gguf) | Q3_K_S | 3 | 30.9 GB| very small, high quality loss |
| [Llama-3-Groq-70B-Tool-Use-Q4_0.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q4_0.gguf) | Q4_0 | 4 | 40.0 GB| legacy; small, very high quality loss - prefer using Q3_K_M |
| [Llama-3-Groq-70B-Tool-Use-Q4_K_M.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q4_K_M.gguf) | Q4_K_M | 4 | 42.5 GB| medium, balanced quality - recommended |
| [Llama-3-Groq-70B-Tool-Use-Q4_K_S.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q4_K_S.gguf) | Q4_K_S | 4 | 40.3 GB| small, greater quality loss |
| [Llama-3-Groq-70B-Tool-Use-Q5_0.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q5_0.gguf) | Q5_0 | 5 | 48.7 GB| legacy; medium, balanced quality - prefer using Q4_K_M |
| [Llama-3-Groq-70B-Tool-Use-Q5_K_M.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q5_K_M.gguf) | Q5_K_M | 5 | 49.9 GB| large, very low quality loss - recommended |
| [Llama-3-Groq-70B-Tool-Use-Q5_K_S.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q5_K_S.gguf) | Q5_K_S | 5 | 48.7 GB| large, low quality loss - recommended |
| [Llama-3-Groq-70B-Tool-Use-Q6_K-00001-of-00002.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q6_K-00001-of-00002.gguf) | Q6_K | 6 | 29.8 GB| very large, extremely low quality loss |
| [Llama-3-Groq-70B-Tool-Use-Q6_K-00002-of-00002.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q6_K-00002-of-00002.gguf) | Q6_K | 6 | 28.0 GB| very large, extremely low quality loss |
| [Llama-3-Groq-70B-Tool-Use-Q8_0-00001-of-00003.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q8_0-00001-of-00003.gguf) | Q8_0 | 8 | 29.8 GB| very large, extremely low quality loss - not recommended |
| [Llama-3-Groq-70B-Tool-Use-Q8_0-00002-of-00003.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q8_0-00002-of-00003.gguf) | Q8_0 | 8 | 29.8 GB| very large, extremely low quality loss - not recommended |
| [Llama-3-Groq-70B-Tool-Use-Q8_0-00003-of-00003.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-Q8_0-00003-of-00003.gguf) | Q8_0 | 8 | 15.4 GB| very large, extremely low quality loss - not recommended |
| [Llama-3-Groq-70B-Tool-Use-f16-00001-of-00005.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-f16-00001-of-00005.gguf) | f16 | 16 | 30.0 GB| |
| [Llama-3-Groq-70B-Tool-Use-f16-00002-of-00005.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-f16-00002-of-00005.gguf) | f16 | 16 | 29.6 GB| |
| [Llama-3-Groq-70B-Tool-Use-f16-00003-of-00005.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-f16-00003-of-00005.gguf) | f16 | 16 | 29.6 GB| |
| [Llama-3-Groq-70B-Tool-Use-f16-00004-of-00005.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-f16-00004-of-00005.gguf) | f16 | 16 | 29.6 GB| |
| [Llama-3-Groq-70B-Tool-Use-f16-00005-of-00005.gguf](https://huggingface.co/second-state/Llama-3-Groq-70B-Tool-Use-GGUF/blob/main/Llama-3-Groq-70B-Tool-Use-f16-00005-of-00005.gguf) | f16 | 16 | 22.2 GB| |
*Quantized with llama.cpp b3463.* |