sharpenb commited on
Commit
c35920e
1 Parent(s): 4cc8e9e

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +11 -11
README.md CHANGED
@@ -32,7 +32,7 @@ tags:
32
  - Contact us and tell us which model to compress next [here](https://www.pruna.ai/contact).
33
  - Request access to easily compress your *own* AI models [here](https://z0halsaff74.typeform.com/pruna-access?typeform-source=www.pruna.ai).
34
  - Read the documentations to know more [here](https://pruna-ai-pruna.readthedocs-hosted.com/en/latest/)
35
- - Join Pruna AI community on Discord [here](https://discord.com/invite/vb6SmA3hxu) to share feedback/suggestions or get help.
36
 
37
  **Frequently Asked Questions**
38
  - ***How does the compression work?*** The model is compressed with GGUF.
@@ -73,7 +73,7 @@ The following clients/libraries will automatically download models for you, prov
73
  * Faraday.dev
74
 
75
  - **Option A** - Downloading in `text-generation-webui`:
76
- - **Step 1**: Under Download Model, you can enter the model repo: PrunaAI/Llama-3-Refueled-GGUF-smashed and below it, a specific filename to download, such as: phi-2.IQ3_M.gguf.
77
  - **Step 2**: Then click Download.
78
 
79
  - **Option B** - Downloading on the command line (including multiple files at once):
@@ -83,14 +83,14 @@ pip3 install huggingface-hub
83
  ```
84
  - **Step 2**: Then you can download any individual model file to the current directory, at high speed, with a command like this:
85
  ```shell
86
- huggingface-cli download PrunaAI/Llama-3-Refueled-GGUF-smashed Llama-3-Refueled.IQ3_M.gguf --local-dir . --local-dir-use-symlinks False
87
  ```
88
  <details>
89
  <summary>More advanced huggingface-cli download usage (click to read)</summary>
90
  Alternatively, you can also download multiple files at once with a pattern:
91
 
92
  ```shell
93
- huggingface-cli download PrunaAI/Llama-3-Refueled-GGUF-smashed --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'
94
  ```
95
 
96
  For more documentation on downloading with `huggingface-cli`, please see: [HF -> Hub Python Library -> Download files -> Download from the CLI](https://huggingface.co/docs/huggingface_hub/guides/download#download-from-the-cli).
@@ -104,7 +104,7 @@ pip3 install hf_transfer
104
  And set environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`:
105
 
106
  ```shell
107
- HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download PrunaAI/Llama-3-Refueled-GGUF-smashed Llama-3-Refueled.IQ3_M.gguf --local-dir . --local-dir-use-symlinks False
108
  ```
109
 
110
  Windows Command Line users: You can set the environment variable by running `set HF_HUB_ENABLE_HF_TRANSFER=1` before the download command.
@@ -119,7 +119,7 @@ Windows Command Line users: You can set the environment variable by running `set
119
  Make sure you are using `llama.cpp` from commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221) or later.
120
 
121
  ```shell
122
- ./main -ngl 35 -m Llama-3-Refueled.IQ3_M.gguf --color -c 32768 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "<s>[INST] {prompt\} [/INST]"
123
  ```
124
 
125
  Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
@@ -180,7 +180,7 @@ You can use GGUF models from Python using the [llama-cpp-python](https://github.
180
 
181
  # Simple inference example
182
  output = llm(
183
- "<s>[INST] {prompt} [/INST]", # Prompt
184
  max_tokens=512, # Generate up to 512 tokens
185
  stop=["</s>"], # Example stop token - not necessarily correct for this specific model! Please check before using.
186
  echo=True # Whether to echo the prompt
@@ -191,11 +191,11 @@ You can use GGUF models from Python using the [llama-cpp-python](https://github.
191
  llm = Llama(model_path="./Llama-3-Refueled.IQ3_M.gguf", chat_format="llama-2") # Set chat_format according to the model you are using
192
  llm.create_chat_completion(
193
  messages = [
194
- {"role": "system", "content": "You are a story writing assistant."},
195
- {
196
  "role": "user",
197
  "content": "Write a story about llamas."
198
- }
199
  ]
200
  )
201
  ```
@@ -218,4 +218,4 @@ The license of the smashed model follows the license of the original model. Plea
218
  ## Want to compress other models?
219
 
220
  - Contact us and tell us which model to compress next [here](https://www.pruna.ai/contact).
221
- - Request access to easily compress your own AI models [here](https://z0halsaff74.typeform.com/pruna-access?typeform-source=www.pruna.ai).
 
32
  - Contact us and tell us which model to compress next [here](https://www.pruna.ai/contact).
33
  - Request access to easily compress your *own* AI models [here](https://z0halsaff74.typeform.com/pruna-access?typeform-source=www.pruna.ai).
34
  - Read the documentations to know more [here](https://pruna-ai-pruna.readthedocs-hosted.com/en/latest/)
35
+ - Join Pruna AI community on Discord [here](https://discord.gg/rskEr4BZJx) to share feedback/suggestions or get help.
36
 
37
  **Frequently Asked Questions**
38
  - ***How does the compression work?*** The model is compressed with GGUF.
 
73
  * Faraday.dev
74
 
75
  - **Option A** - Downloading in `text-generation-webui`:
76
+ - **Step 1**: Under Download Model, you can enter the model repo: refuelai-Llama-3-Refueled-GGUF-smashed and below it, a specific filename to download, such as: phi-2.IQ3_M.gguf.
77
  - **Step 2**: Then click Download.
78
 
79
  - **Option B** - Downloading on the command line (including multiple files at once):
 
83
  ```
84
  - **Step 2**: Then you can download any individual model file to the current directory, at high speed, with a command like this:
85
  ```shell
86
+ huggingface-cli download refuelai-Llama-3-Refueled-GGUF-smashed Llama-3-Refueled.IQ3_M.gguf --local-dir . --local-dir-use-symlinks False
87
  ```
88
  <details>
89
  <summary>More advanced huggingface-cli download usage (click to read)</summary>
90
  Alternatively, you can also download multiple files at once with a pattern:
91
 
92
  ```shell
93
+ huggingface-cli download refuelai-Llama-3-Refueled-GGUF-smashed --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'
94
  ```
95
 
96
  For more documentation on downloading with `huggingface-cli`, please see: [HF -> Hub Python Library -> Download files -> Download from the CLI](https://huggingface.co/docs/huggingface_hub/guides/download#download-from-the-cli).
 
104
  And set environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`:
105
 
106
  ```shell
107
+ HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download refuelai-Llama-3-Refueled-GGUF-smashed Llama-3-Refueled.IQ3_M.gguf --local-dir . --local-dir-use-symlinks False
108
  ```
109
 
110
  Windows Command Line users: You can set the environment variable by running `set HF_HUB_ENABLE_HF_TRANSFER=1` before the download command.
 
119
  Make sure you are using `llama.cpp` from commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221) or later.
120
 
121
  ```shell
122
+ ./main -ngl 35 -m Llama-3-Refueled.IQ3_M.gguf --color -c 32768 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "<s>[INST] {{prompt\}} [/INST]"
123
  ```
124
 
125
  Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
 
180
 
181
  # Simple inference example
182
  output = llm(
183
+ "<s>[INST] {{prompt}} [/INST]", # Prompt
184
  max_tokens=512, # Generate up to 512 tokens
185
  stop=["</s>"], # Example stop token - not necessarily correct for this specific model! Please check before using.
186
  echo=True # Whether to echo the prompt
 
191
  llm = Llama(model_path="./Llama-3-Refueled.IQ3_M.gguf", chat_format="llama-2") # Set chat_format according to the model you are using
192
  llm.create_chat_completion(
193
  messages = [
194
+ {{"role": "system", "content": "You are a story writing assistant."}},
195
+ {{
196
  "role": "user",
197
  "content": "Write a story about llamas."
198
+ }}
199
  ]
200
  )
201
  ```
 
218
  ## Want to compress other models?
219
 
220
  - Contact us and tell us which model to compress next [here](https://www.pruna.ai/contact).
221
+ - Request access to easily compress your own AI models [here](https://z0halsaff74.typeform.com/pruna-access?typeform-source=www.pruna.ai).