sylwia-kuros commited on
Commit
b9c169b
1 Parent(s): 4d807d6

Add disclaimer

Browse files
Files changed (1) hide show
  1. README.md +74 -70
README.md CHANGED
@@ -1,70 +1,74 @@
1
- ---
2
- license: apache-2.0
3
- language:
4
- - en
5
- ---
6
-
7
- # Mixtral-8x7b-Instruct-v0.1-int8-ov
8
-
9
- * Model creator: [Mistral AI](https://huggingface.co/mistralai)
10
- * Original model: [Mixtral 8X7B Instruct v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)
11
-
12
- ## Description
13
-
14
- This is [Mixtral-8x7b-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model converted to the [OpenVINO™ IR](https://docs.openvino.ai/2024/documentation/openvino-ir-format.html) (Intermediate Representation) format with weights compressed to INT8 by [NNCF](https://github.com/openvinotoolkit/nncf).
15
-
16
- ## Quantization Parameters
17
-
18
- Weight compression was performed using `nncf.compress_weights` with the following parameters:
19
-
20
- * mode: **INT8_ASYM**
21
-
22
- For more information on quantization, check the [OpenVINO model optimization guide](https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html).
23
-
24
- ## Compatibility
25
-
26
- The provided OpenVINO™ IR model is compatible with:
27
-
28
- * OpenVINO version 2024.0.0 and higher
29
- * Optimum Intel 1.16.0 and higher
30
-
31
- ## Running Model Inference
32
-
33
- 1. Install packages required for using [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) integration with the OpenVINO backend:
34
-
35
- ```
36
- pip install optimum[openvino]
37
- ```
38
-
39
- 2. Run model inference:
40
-
41
- ```
42
- from transformers import AutoTokenizer
43
- from optimum.intel.openvino import OVModelForCausalLM
44
-
45
- model_id = "OpenVINO/mixtral-8x7b-instruct-v0.1-int8-ov"
46
- tokenizer = AutoTokenizer.from_pretrained(model_id)
47
- model = OVModelForCausalLM.from_pretrained(model_id)
48
-
49
-
50
- messages = [
51
- {"role": "user", "content": "What is your favourite condiment?"},
52
- {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
53
- {"role": "user", "content": "Do you have mayonnaise recipes?"}
54
- ]
55
-
56
- inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
57
-
58
- outputs = model.generate(inputs, max_new_tokens=20)
59
- print(tokenizer.decode(outputs[0], skip_special_tokens=True))
60
- ```
61
-
62
- For more examples and possible optimizations, refer to the [OpenVINO Large Language Model Inference Guide](https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html).
63
-
64
- ## Limitations
65
-
66
- Check the original model card for [limitations](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1#limitations).
67
-
68
- ## Legal information
69
-
70
- The original model is distributed under [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) license. More details can be found in [original model card](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1).
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ ---
6
+
7
+ # Mixtral-8x7b-Instruct-v0.1-int8-ov
8
+
9
+ * Model creator: [Mistral AI](https://huggingface.co/mistralai)
10
+ * Original model: [Mixtral 8X7B Instruct v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)
11
+
12
+ ## Description
13
+
14
+ This is [Mixtral-8x7b-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model converted to the [OpenVINO™ IR](https://docs.openvino.ai/2024/documentation/openvino-ir-format.html) (Intermediate Representation) format with weights compressed to INT8 by [NNCF](https://github.com/openvinotoolkit/nncf).
15
+
16
+ ## Quantization Parameters
17
+
18
+ Weight compression was performed using `nncf.compress_weights` with the following parameters:
19
+
20
+ * mode: **INT8_ASYM**
21
+
22
+ For more information on quantization, check the [OpenVINO model optimization guide](https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html).
23
+
24
+ ## Compatibility
25
+
26
+ The provided OpenVINO™ IR model is compatible with:
27
+
28
+ * OpenVINO version 2024.0.0 and higher
29
+ * Optimum Intel 1.16.0 and higher
30
+
31
+ ## Running Model Inference
32
+
33
+ 1. Install packages required for using [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) integration with the OpenVINO backend:
34
+
35
+ ```
36
+ pip install optimum[openvino]
37
+ ```
38
+
39
+ 2. Run model inference:
40
+
41
+ ```
42
+ from transformers import AutoTokenizer
43
+ from optimum.intel.openvino import OVModelForCausalLM
44
+
45
+ model_id = "OpenVINO/mixtral-8x7b-instruct-v0.1-int8-ov"
46
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
47
+ model = OVModelForCausalLM.from_pretrained(model_id)
48
+
49
+
50
+ messages = [
51
+ {"role": "user", "content": "What is your favourite condiment?"},
52
+ {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
53
+ {"role": "user", "content": "Do you have mayonnaise recipes?"}
54
+ ]
55
+
56
+ inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
57
+
58
+ outputs = model.generate(inputs, max_new_tokens=20)
59
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
60
+ ```
61
+
62
+ For more examples and possible optimizations, refer to the [OpenVINO Large Language Model Inference Guide](https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html).
63
+
64
+ ## Limitations
65
+
66
+ Check the original model card for [limitations](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1#limitations).
67
+
68
+ ## Legal information
69
+
70
+ The original model is distributed under [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) license. More details can be found in [original model card](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1).
71
+
72
+ ## Disclaimer
73
+
74
+ Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See [Intel’s Global Human Rights Principles](https://www.intel.com/content/dam/www/central-libraries/us/en/documents/policy-human-rights.pdf). Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.