yhavinga commited on
Commit
e974e27
1 Parent(s): 3bda927

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -30,18 +30,18 @@ The tokenizer was trained using the `spm_train` command with the following setti
30
 
31
  ## Installation
32
  To use the Dutch-Llama Tokenizer, ensure you have Python 3.10.12 or later installed. Then, install the Transformers library from Hugging Face:
33
- ```
34
  pip install transformers
35
  ```
36
 
37
  ## Usage
38
  First, import the `AutoTokenizer` from the Transformers library and load the Dutch-Llama Tokenizer:
39
- ```
40
  from transformers import AutoTokenizer
41
  tokenizer = AutoTokenizer.from_pretrained("yhavinga/dutch-llama-tokenizer")
42
  ```
43
  To tokenize text, use the `tokenizer.tokenize` method. For converting tokens to IDs and decoding them back to text, use `tokenizer.convert_tokens_to_ids` and `tokenizer.decode` respectively:
44
- ```
45
  # Example text
46
  text = "Steenvliegen of oevervliegen[2] (Plecoptera) 鍗庝负鍙戝竷Mate60鎵嬫満"
47
 
 
30
 
31
  ## Installation
32
  To use the Dutch-Llama Tokenizer, ensure you have Python 3.10.12 or later installed. Then, install the Transformers library from Hugging Face:
33
+ ```shell
34
  pip install transformers
35
  ```
36
 
37
  ## Usage
38
  First, import the `AutoTokenizer` from the Transformers library and load the Dutch-Llama Tokenizer:
39
+ ```python
40
  from transformers import AutoTokenizer
41
  tokenizer = AutoTokenizer.from_pretrained("yhavinga/dutch-llama-tokenizer")
42
  ```
43
  To tokenize text, use the `tokenizer.tokenize` method. For converting tokens to IDs and decoding them back to text, use `tokenizer.convert_tokens_to_ids` and `tokenizer.decode` respectively:
44
+ ```python
45
  # Example text
46
  text = "Steenvliegen of oevervliegen[2] (Plecoptera) 鍗庝负鍙戝竷Mate60鎵嬫満"
47