title: Autoeval config
comments: _The documentation is not available anymore as the PR was closed or merged._ Related to: https://github.com/huggingface/autonlp-backend/issues/414 and https://github.com/huggingface/autonlp-backend/issues/424 The tests are failing due to the changed metadata:
  
  ```
  got an unexpected keyword argument 'train-eval-index'
  ```
  
  I think you can fix this by updating the `DatasetMetadata` class and implementing an appropriate `validate_train_eval_index()` function
  
  @lhoestq we are working with an arbitrary set of tags for `autoeval config`. See https://github.com/huggingface/autonlp-backend/issues/414
  I need to add a validator function though for the tests to pass. Our set is not well-defined as in the rest https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources. What's a workaround for this? On the question of validating the `train-eval-index` metadata, I think the simplest approach would be to validate that the required fields exist and not worry about their values (which are open-ended).
  
  For me, the required fields include:
  
  * `config`
  * `task`
  * `task_id`
  * `splits` (train / validation / eval)
  * `col_mapping`
  * `metrics` (checking that each one has `type`, `name`) 
  
  Here I'm using the spec defined in https://github.com/huggingface/autonlp-backend/issues/414 as a guide.
  
  WDYT @lhoestq ? Makes sense ! Currently the metadata type validator doesn't support subfields - let me open a PR to add it I ended up improving the metadata validation in this PR x)
  
  In particular:
  - I added support YAML keys with dashes instead of underscores for `train-eval-index`
  - I added `train-eval-index` validation  with `validate_train_eval_index`. It does nothing fancy, it just checks that it is a list if it exists in the YAML, but feel free to improve it if you want
  
  Let me know if it sounds good to you ! I think we can improve `validate_train_eval_index` in another PR Come on windows... I didn't do anything advanced...
  
  Anyway, will try to fix this when I get back home x) > Come on windows... I didn't do anything advanced...
  > 
  > Anyway, will try to fix this when I get back home x)
  
  Hehe, thanks! Thanks, @lhoestq this is great!  Did I just fix it for windows and now it fails on linux ? xD > Did I just fix it for windows and now it fails on linux ? xD
  
  Looks like the Heisenberg uncertainty principle is at play here - you cannot simultaneously have unit tests passing in both Linux and Windows 😅  The worst is that the tests pass locally both on my windows and my linux x) Ok fixed it, the issue came from python 3.6 that doesn't return the right `__origin__` for Dict and List types > Alright thanks for adding the first Autoeval config ! :D
  
  Woohoo! Thank you so much 🤗  This is cool!
body: Added autoeval config to imdb as pilot