tomaarsen HF staff commited on
Commit
e4dbe50
1 Parent(s): beac42d

Remove ` (default)` from MTEB scores caused by an MTEB bug

Browse files

Hello!

## Pull Request overview
* Remove ` (default)` from MTEB scores

## Details
Recently, a bug on MTEB caused the dataset names to be updated from e.g. "ArguAna" to include the config (usually "default", but sometimes a language) in the dataset name. This resulted in a discrepancy between the expected dataset names and the true dataset names in e.g. this model card. This PR should fix it. I've verified that it should work by copying this model card to a private repository and checking that MTEB correctly parses the updated one.

Apologies for the inconvenience. Once we've successfully put this model on MTEB, I'd be glad to help announce the model release on LI/X.

- Tom Aarsen

Files changed (1) hide show
  1. README.md +27 -27
README.md CHANGED
@@ -14,7 +14,7 @@ model-index:
14
  results:
15
  - dataset:
16
  config: default
17
- name: MTEB ArguAna (default)
18
  revision: c22ab2a51041ffd869aaddef7af8d8215647e41a
19
  split: test
20
  type: mteb/arguana
@@ -305,7 +305,7 @@ model-index:
305
  type: Retrieval
306
  - dataset:
307
  config: default
308
- name: MTEB CQADupstackAndroidRetrieval (default)
309
  revision: f46a197baaae43b4f621051089b82a364682dfeb
310
  split: test
311
  type: mteb/cqadupstack-android
@@ -596,7 +596,7 @@ model-index:
596
  type: Retrieval
597
  - dataset:
598
  config: default
599
- name: MTEB CQADupstackEnglishRetrieval (default)
600
  revision: ad9991cb51e31e31e430383c75ffb2885547b5f0
601
  split: test
602
  type: mteb/cqadupstack-english
@@ -887,7 +887,7 @@ model-index:
887
  type: Retrieval
888
  - dataset:
889
  config: default
890
- name: MTEB CQADupstackGamingRetrieval (default)
891
  revision: 4885aa143210c98657558c04aaf3dc47cfb54340
892
  split: test
893
  type: mteb/cqadupstack-gaming
@@ -1178,7 +1178,7 @@ model-index:
1178
  type: Retrieval
1179
  - dataset:
1180
  config: default
1181
- name: MTEB CQADupstackGisRetrieval (default)
1182
  revision: 5003b3064772da1887988e05400cf3806fe491f2
1183
  split: test
1184
  type: mteb/cqadupstack-gis
@@ -1469,7 +1469,7 @@ model-index:
1469
  type: Retrieval
1470
  - dataset:
1471
  config: default
1472
- name: MTEB CQADupstackMathematicaRetrieval (default)
1473
  revision: 90fceea13679c63fe563ded68f3b6f06e50061de
1474
  split: test
1475
  type: mteb/cqadupstack-mathematica
@@ -1760,7 +1760,7 @@ model-index:
1760
  type: Retrieval
1761
  - dataset:
1762
  config: default
1763
- name: MTEB CQADupstackPhysicsRetrieval (default)
1764
  revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4
1765
  split: test
1766
  type: mteb/cqadupstack-physics
@@ -2051,7 +2051,7 @@ model-index:
2051
  type: Retrieval
2052
  - dataset:
2053
  config: default
2054
- name: MTEB CQADupstackProgrammersRetrieval (default)
2055
  revision: 6184bc1440d2dbc7612be22b50686b8826d22b32
2056
  split: test
2057
  type: mteb/cqadupstack-programmers
@@ -2342,7 +2342,7 @@ model-index:
2342
  type: Retrieval
2343
  - dataset:
2344
  config: default
2345
- name: MTEB CQADupstackRetrieval (default)
2346
  revision: CQADupstackRetrieval is a combined dataset
2347
  split: test
2348
  type: mteb/cqadupstack
@@ -2355,7 +2355,7 @@ model-index:
2355
  type: Retrieval
2356
  - dataset:
2357
  config: default
2358
- name: MTEB CQADupstackStatsRetrieval (default)
2359
  revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a
2360
  split: test
2361
  type: mteb/cqadupstack-stats
@@ -2646,7 +2646,7 @@ model-index:
2646
  type: Retrieval
2647
  - dataset:
2648
  config: default
2649
- name: MTEB CQADupstackTexRetrieval (default)
2650
  revision: 46989137a86843e03a6195de44b09deda022eec7
2651
  split: test
2652
  type: mteb/cqadupstack-tex
@@ -2937,7 +2937,7 @@ model-index:
2937
  type: Retrieval
2938
  - dataset:
2939
  config: default
2940
- name: MTEB CQADupstackUnixRetrieval (default)
2941
  revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53
2942
  split: test
2943
  type: mteb/cqadupstack-unix
@@ -3228,7 +3228,7 @@ model-index:
3228
  type: Retrieval
3229
  - dataset:
3230
  config: default
3231
- name: MTEB CQADupstackWebmastersRetrieval (default)
3232
  revision: 160c094312a0e1facb97e55eeddb698c0abe3571
3233
  split: test
3234
  type: mteb/cqadupstack-webmasters
@@ -3519,7 +3519,7 @@ model-index:
3519
  type: Retrieval
3520
  - dataset:
3521
  config: default
3522
- name: MTEB CQADupstackWordpressRetrieval (default)
3523
  revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4
3524
  split: test
3525
  type: mteb/cqadupstack-wordpress
@@ -3810,7 +3810,7 @@ model-index:
3810
  type: Retrieval
3811
  - dataset:
3812
  config: default
3813
- name: MTEB ClimateFEVER (default)
3814
  revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380
3815
  split: test
3816
  type: mteb/climate-fever
@@ -4101,7 +4101,7 @@ model-index:
4101
  type: Retrieval
4102
  - dataset:
4103
  config: default
4104
- name: MTEB DBPedia (default)
4105
  revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659
4106
  split: test
4107
  type: mteb/dbpedia
@@ -4392,7 +4392,7 @@ model-index:
4392
  type: Retrieval
4393
  - dataset:
4394
  config: default
4395
- name: MTEB FEVER (default)
4396
  revision: bea83ef9e8fb933d90a2f1d5515737465d613e12
4397
  split: test
4398
  type: mteb/fever
@@ -4683,7 +4683,7 @@ model-index:
4683
  type: Retrieval
4684
  - dataset:
4685
  config: default
4686
- name: MTEB FiQA2018 (default)
4687
  revision: 27a168819829fe9bcd655c2df245fb19452e8e06
4688
  split: test
4689
  type: mteb/fiqa
@@ -4974,7 +4974,7 @@ model-index:
4974
  type: Retrieval
4975
  - dataset:
4976
  config: default
4977
- name: MTEB HotpotQA (default)
4978
  revision: ab518f4d6fcca38d87c25209f94beba119d02014
4979
  split: test
4980
  type: mteb/hotpotqa
@@ -5265,7 +5265,7 @@ model-index:
5265
  type: Retrieval
5266
  - dataset:
5267
  config: default
5268
- name: MTEB MSMARCO (default)
5269
  revision: c5a29a104738b98a9e76336939199e264163d4a0
5270
  split: dev
5271
  type: mteb/msmarco
@@ -5556,7 +5556,7 @@ model-index:
5556
  type: Retrieval
5557
  - dataset:
5558
  config: default
5559
- name: MTEB NFCorpus (default)
5560
  revision: ec0fa4fe99da2ff19ca1214b7966684033a58814
5561
  split: test
5562
  type: mteb/nfcorpus
@@ -5847,7 +5847,7 @@ model-index:
5847
  type: Retrieval
5848
  - dataset:
5849
  config: default
5850
- name: MTEB NQ (default)
5851
  revision: b774495ed302d8c44a3a7ea25c90dbce03968f31
5852
  split: test
5853
  type: mteb/nq
@@ -6138,7 +6138,7 @@ model-index:
6138
  type: Retrieval
6139
  - dataset:
6140
  config: default
6141
- name: MTEB QuoraRetrieval (default)
6142
  revision: e4e08e0b7dbe3c8700f0daef558ff32256715259
6143
  split: test
6144
  type: mteb/quora
@@ -6429,7 +6429,7 @@ model-index:
6429
  type: Retrieval
6430
  - dataset:
6431
  config: default
6432
- name: MTEB SCIDOCS (default)
6433
  revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88
6434
  split: test
6435
  type: mteb/scidocs
@@ -6720,7 +6720,7 @@ model-index:
6720
  type: Retrieval
6721
  - dataset:
6722
  config: default
6723
- name: MTEB SciFact (default)
6724
  revision: 0228b52cf27578f30900b9e5271d331663a030d7
6725
  split: test
6726
  type: mteb/scifact
@@ -7011,7 +7011,7 @@ model-index:
7011
  type: Retrieval
7012
  - dataset:
7013
  config: default
7014
- name: MTEB TRECCOVID (default)
7015
  revision: bb9466bac8153a0349341eb1b22e06409e78ef4e
7016
  split: test
7017
  type: mteb/trec-covid
@@ -7302,7 +7302,7 @@ model-index:
7302
  type: Retrieval
7303
  - dataset:
7304
  config: default
7305
- name: MTEB Touche2020 (default)
7306
  revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f
7307
  split: test
7308
  type: mteb/touche2020
 
14
  results:
15
  - dataset:
16
  config: default
17
+ name: MTEB ArguAna
18
  revision: c22ab2a51041ffd869aaddef7af8d8215647e41a
19
  split: test
20
  type: mteb/arguana
 
305
  type: Retrieval
306
  - dataset:
307
  config: default
308
+ name: MTEB CQADupstackAndroidRetrieval
309
  revision: f46a197baaae43b4f621051089b82a364682dfeb
310
  split: test
311
  type: mteb/cqadupstack-android
 
596
  type: Retrieval
597
  - dataset:
598
  config: default
599
+ name: MTEB CQADupstackEnglishRetrieval
600
  revision: ad9991cb51e31e31e430383c75ffb2885547b5f0
601
  split: test
602
  type: mteb/cqadupstack-english
 
887
  type: Retrieval
888
  - dataset:
889
  config: default
890
+ name: MTEB CQADupstackGamingRetrieval
891
  revision: 4885aa143210c98657558c04aaf3dc47cfb54340
892
  split: test
893
  type: mteb/cqadupstack-gaming
 
1178
  type: Retrieval
1179
  - dataset:
1180
  config: default
1181
+ name: MTEB CQADupstackGisRetrieval
1182
  revision: 5003b3064772da1887988e05400cf3806fe491f2
1183
  split: test
1184
  type: mteb/cqadupstack-gis
 
1469
  type: Retrieval
1470
  - dataset:
1471
  config: default
1472
+ name: MTEB CQADupstackMathematicaRetrieval
1473
  revision: 90fceea13679c63fe563ded68f3b6f06e50061de
1474
  split: test
1475
  type: mteb/cqadupstack-mathematica
 
1760
  type: Retrieval
1761
  - dataset:
1762
  config: default
1763
+ name: MTEB CQADupstackPhysicsRetrieval
1764
  revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4
1765
  split: test
1766
  type: mteb/cqadupstack-physics
 
2051
  type: Retrieval
2052
  - dataset:
2053
  config: default
2054
+ name: MTEB CQADupstackProgrammersRetrieval
2055
  revision: 6184bc1440d2dbc7612be22b50686b8826d22b32
2056
  split: test
2057
  type: mteb/cqadupstack-programmers
 
2342
  type: Retrieval
2343
  - dataset:
2344
  config: default
2345
+ name: MTEB CQADupstackRetrieval
2346
  revision: CQADupstackRetrieval is a combined dataset
2347
  split: test
2348
  type: mteb/cqadupstack
 
2355
  type: Retrieval
2356
  - dataset:
2357
  config: default
2358
+ name: MTEB CQADupstackStatsRetrieval
2359
  revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a
2360
  split: test
2361
  type: mteb/cqadupstack-stats
 
2646
  type: Retrieval
2647
  - dataset:
2648
  config: default
2649
+ name: MTEB CQADupstackTexRetrieval
2650
  revision: 46989137a86843e03a6195de44b09deda022eec7
2651
  split: test
2652
  type: mteb/cqadupstack-tex
 
2937
  type: Retrieval
2938
  - dataset:
2939
  config: default
2940
+ name: MTEB CQADupstackUnixRetrieval
2941
  revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53
2942
  split: test
2943
  type: mteb/cqadupstack-unix
 
3228
  type: Retrieval
3229
  - dataset:
3230
  config: default
3231
+ name: MTEB CQADupstackWebmastersRetrieval
3232
  revision: 160c094312a0e1facb97e55eeddb698c0abe3571
3233
  split: test
3234
  type: mteb/cqadupstack-webmasters
 
3519
  type: Retrieval
3520
  - dataset:
3521
  config: default
3522
+ name: MTEB CQADupstackWordpressRetrieval
3523
  revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4
3524
  split: test
3525
  type: mteb/cqadupstack-wordpress
 
3810
  type: Retrieval
3811
  - dataset:
3812
  config: default
3813
+ name: MTEB ClimateFEVER
3814
  revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380
3815
  split: test
3816
  type: mteb/climate-fever
 
4101
  type: Retrieval
4102
  - dataset:
4103
  config: default
4104
+ name: MTEB DBPedia
4105
  revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659
4106
  split: test
4107
  type: mteb/dbpedia
 
4392
  type: Retrieval
4393
  - dataset:
4394
  config: default
4395
+ name: MTEB FEVER
4396
  revision: bea83ef9e8fb933d90a2f1d5515737465d613e12
4397
  split: test
4398
  type: mteb/fever
 
4683
  type: Retrieval
4684
  - dataset:
4685
  config: default
4686
+ name: MTEB FiQA2018
4687
  revision: 27a168819829fe9bcd655c2df245fb19452e8e06
4688
  split: test
4689
  type: mteb/fiqa
 
4974
  type: Retrieval
4975
  - dataset:
4976
  config: default
4977
+ name: MTEB HotpotQA
4978
  revision: ab518f4d6fcca38d87c25209f94beba119d02014
4979
  split: test
4980
  type: mteb/hotpotqa
 
5265
  type: Retrieval
5266
  - dataset:
5267
  config: default
5268
+ name: MTEB MSMARCO
5269
  revision: c5a29a104738b98a9e76336939199e264163d4a0
5270
  split: dev
5271
  type: mteb/msmarco
 
5556
  type: Retrieval
5557
  - dataset:
5558
  config: default
5559
+ name: MTEB NFCorpus
5560
  revision: ec0fa4fe99da2ff19ca1214b7966684033a58814
5561
  split: test
5562
  type: mteb/nfcorpus
 
5847
  type: Retrieval
5848
  - dataset:
5849
  config: default
5850
+ name: MTEB NQ
5851
  revision: b774495ed302d8c44a3a7ea25c90dbce03968f31
5852
  split: test
5853
  type: mteb/nq
 
6138
  type: Retrieval
6139
  - dataset:
6140
  config: default
6141
+ name: MTEB QuoraRetrieval
6142
  revision: e4e08e0b7dbe3c8700f0daef558ff32256715259
6143
  split: test
6144
  type: mteb/quora
 
6429
  type: Retrieval
6430
  - dataset:
6431
  config: default
6432
+ name: MTEB SCIDOCS
6433
  revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88
6434
  split: test
6435
  type: mteb/scidocs
 
6720
  type: Retrieval
6721
  - dataset:
6722
  config: default
6723
+ name: MTEB SciFact
6724
  revision: 0228b52cf27578f30900b9e5271d331663a030d7
6725
  split: test
6726
  type: mteb/scifact
 
7011
  type: Retrieval
7012
  - dataset:
7013
  config: default
7014
+ name: MTEB TRECCOVID
7015
  revision: bb9466bac8153a0349341eb1b22e06409e78ef4e
7016
  split: test
7017
  type: mteb/trec-covid
 
7302
  type: Retrieval
7303
  - dataset:
7304
  config: default
7305
+ name: MTEB Touche2020
7306
  revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f
7307
  split: test
7308
  type: mteb/touche2020