Update README.md
Browse files
README.md
CHANGED
@@ -30,18 +30,18 @@ The tokenizer was trained using the `spm_train` command with the following setti
|
|
30 |
|
31 |
## Installation
|
32 |
To use the Dutch-Llama Tokenizer, ensure you have Python 3.10.12 or later installed. Then, install the Transformers library from Hugging Face:
|
33 |
-
```
|
34 |
pip install transformers
|
35 |
```
|
36 |
|
37 |
## Usage
|
38 |
First, import the `AutoTokenizer` from the Transformers library and load the Dutch-Llama Tokenizer:
|
39 |
-
```
|
40 |
from transformers import AutoTokenizer
|
41 |
tokenizer = AutoTokenizer.from_pretrained("yhavinga/dutch-llama-tokenizer")
|
42 |
```
|
43 |
To tokenize text, use the `tokenizer.tokenize` method. For converting tokens to IDs and decoding them back to text, use `tokenizer.convert_tokens_to_ids` and `tokenizer.decode` respectively:
|
44 |
-
```
|
45 |
# Example text
|
46 |
text = "Steenvliegen of oevervliegen[2] (Plecoptera) 鍗庝负鍙戝竷Mate60鎵嬫満"
|
47 |
|
|
|
30 |
|
31 |
## Installation
|
32 |
To use the Dutch-Llama Tokenizer, ensure you have Python 3.10.12 or later installed. Then, install the Transformers library from Hugging Face:
|
33 |
+
```shell
|
34 |
pip install transformers
|
35 |
```
|
36 |
|
37 |
## Usage
|
38 |
First, import the `AutoTokenizer` from the Transformers library and load the Dutch-Llama Tokenizer:
|
39 |
+
```python
|
40 |
from transformers import AutoTokenizer
|
41 |
tokenizer = AutoTokenizer.from_pretrained("yhavinga/dutch-llama-tokenizer")
|
42 |
```
|
43 |
To tokenize text, use the `tokenizer.tokenize` method. For converting tokens to IDs and decoding them back to text, use `tokenizer.convert_tokens_to_ids` and `tokenizer.decode` respectively:
|
44 |
+
```python
|
45 |
# Example text
|
46 |
text = "Steenvliegen of oevervliegen[2] (Plecoptera) 鍗庝负鍙戝竷Mate60鎵嬫満"
|
47 |
|