Spaces:
Runtime error
Runtime error
If you are using a custom dataset, please provide your dataset definition in the following format in `dataset_info.json`. | |
```json | |
"dataset_name": { | |
"hf_hub_url": "the name of the dataset repository on the Hugging Face hub. (if specified, ignore script_url and file_name)", | |
"ms_hub_url": "the name of the dataset repository on the ModelScope hub. (if specified, ignore script_url and file_name)", | |
"script_url": "the name of the directory containing a dataset loading script. (if specified, ignore file_name)", | |
"file_name": "the name of the dataset file in this directory. (required if above are not specified)", | |
"file_sha1": "the SHA-1 hash value of the dataset file. (optional, does not affect training)", | |
"subset": "the name of the subset. (optional, default: None)", | |
"folder": "the name of the folder of the dataset repository on the Hugging Face hub. (optional, default: None)", | |
"ranking": "whether the dataset is a preference dataset or not. (default: false)", | |
"formatting": "the format of the dataset. (optional, default: alpaca, can be chosen from {alpaca, sharegpt})", | |
"columns": { | |
"prompt": "the column name in the dataset containing the prompts. (default: instruction)", | |
"query": "the column name in the dataset containing the queries. (default: input)", | |
"response": "the column name in the dataset containing the responses. (default: output)", | |
"history": "the column name in the dataset containing the histories. (default: None)", | |
"messages": "the column name in the dataset containing the messages. (default: conversations)", | |
"system": "the column name in the dataset containing the system prompts. (default: None)", | |
"tools": "the column name in the dataset containing the tool description. (default: None)" | |
}, | |
"tags": { | |
"role_tag": "the key in the message represents the identity. (default: from)", | |
"content_tag": "the key in the message represents the content. (default: value)", | |
"user_tag": "the value of the role_tag represents the user. (default: human)", | |
"assistant_tag": "the value of the role_tag represents the assistant. (default: gpt)", | |
"observation_tag": "the value of the role_tag represents the tool results. (default: observation)", | |
"function_tag": "the value of the role_tag represents the function call. (default: function_call)" | |
} | |
} | |
``` | |
Given above, you can use the custom dataset via specifying `--dataset dataset_name`. | |
Currently we support dataset in **alpaca** or **sharegpt** format, the dataset in alpaca format should follow the below format: | |
```json | |
[ | |
{ | |
"instruction": "user instruction (required)", | |
"input": "user input (optional)", | |
"output": "model response (required)", | |
"system": "system prompt (optional)", | |
"history": [ | |
["user instruction in the first round (optional)", "model response in the first round (optional)"], | |
["user instruction in the second round (optional)", "model response in the second round (optional)"] | |
] | |
} | |
] | |
``` | |
Regarding the above dataset, the `columns` in `dataset_info.json` should be: | |
```json | |
"dataset_name": { | |
"columns": { | |
"prompt": "instruction", | |
"query": "input", | |
"response": "output", | |
"system": "system", | |
"history": "history" | |
} | |
} | |
``` | |
where the `prompt` and `response` columns should contain non-empty values, represent instruction and response respectively. The `query` column will be concatenated with the `prompt` column and used as input for the model. | |
The `system` column will be used as the system prompt in the template. The `history` column is a list consisting string tuples representing query-response pairs in history. Note that the responses **in each round will be used for training**. | |
For the pre-training datasets, only the `prompt` column will be used for training. | |
For the preference datasets, the `response` column should be a string list whose length is 2, with the preferred answers appearing first, for example: | |
```json | |
{ | |
"instruction": "user instruction", | |
"input": "user input", | |
"output": [ | |
"chosen answer", | |
"rejected answer" | |
] | |
} | |
``` | |
The dataset in sharegpt format should follow the below format: | |
```json | |
[ | |
{ | |
"conversations": [ | |
{ | |
"from": "human", | |
"value": "user instruction" | |
}, | |
{ | |
"from": "gpt", | |
"value": "model response" | |
} | |
], | |
"system": "system prompt (optional)", | |
"tools": "tool description (optional)" | |
} | |
] | |
``` | |
Regarding the above dataset, the `columns` in `dataset_info.json` should be: | |
```json | |
"dataset_name": { | |
"columns": { | |
"messages": "conversations", | |
"system": "system", | |
"tools": "tools" | |
}, | |
"tags": { | |
"role_tag": "from", | |
"content_tag": "value" | |
} | |
} | |
``` | |
where the `messages` column should be a list whose length is even, and follow the `u/a/u/a/u/a` order. | |
Pre-training datasets and preference datasets are incompatible with the sharegpt format yet. | |