neofung commited on
Commit
c05fab5
·
verified ·
1 Parent(s): 55c3aec

Update README.md

Browse files

update mteb metric

Files changed (1) hide show
  1. README.md +1176 -0
README.md CHANGED
@@ -6,4 +6,1180 @@ tags:
6
  - sentence-transformers
7
  - sentence-similarity
8
  - mteb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9
  ---
 
6
  - sentence-transformers
7
  - sentence-similarity
8
  - mteb
9
+ model-index:
10
+ - name: zh
11
+ results:
12
+ - task:
13
+ type: STS
14
+ dataset:
15
+ type: C-MTEB/AFQMC
16
+ name: MTEB AFQMC
17
+ config: default
18
+ split: validation
19
+ revision: b44c3b011063adb25877c13823db83bb193913c4
20
+ metrics:
21
+ - type: cos_sim_pearson
22
+ value: 36.298796333105045
23
+ - type: cos_sim_spearman
24
+ value: 37.341572623120264
25
+ - type: euclidean_pearson
26
+ value: 36.64243583928722
27
+ - type: euclidean_spearman
28
+ value: 37.34155251464738
29
+ - type: manhattan_pearson
30
+ value: 36.619985504965335
31
+ - type: manhattan_spearman
32
+ value: 37.33234474856907
33
+ - task:
34
+ type: STS
35
+ dataset:
36
+ type: C-MTEB/ATEC
37
+ name: MTEB ATEC
38
+ config: default
39
+ split: test
40
+ revision: 0f319b1142f28d00e055a6770f3f726ae9b7d865
41
+ metrics:
42
+ - type: cos_sim_pearson
43
+ value: 39.479909222662606
44
+ - type: cos_sim_spearman
45
+ value: 41.931326522020626
46
+ - type: euclidean_pearson
47
+ value: 42.96034449758286
48
+ - type: euclidean_spearman
49
+ value: 41.9313332762864
50
+ - type: manhattan_pearson
51
+ value: 42.91465081487226
52
+ - type: manhattan_spearman
53
+ value: 41.891856959411506
54
+ - task:
55
+ type: Classification
56
+ dataset:
57
+ type: mteb/amazon_reviews_multi
58
+ name: MTEB AmazonReviewsClassification (zh)
59
+ config: zh
60
+ split: test
61
+ revision: 1399c76144fd37290681b995c656ef9b2e06e26d
62
+ metrics:
63
+ - type: accuracy
64
+ value: 47.553999999999995
65
+ - type: f1
66
+ value: 44.225487078758015
67
+ - task:
68
+ type: STS
69
+ dataset:
70
+ type: C-MTEB/BQ
71
+ name: MTEB BQ
72
+ config: default
73
+ split: test
74
+ revision: e3dda5e115e487b39ec7e618c0c6a29137052a55
75
+ metrics:
76
+ - type: cos_sim_pearson
77
+ value: 67.41540794294842
78
+ - type: cos_sim_spearman
79
+ value: 71.19680444103656
80
+ - type: euclidean_pearson
81
+ value: 69.59374493550702
82
+ - type: euclidean_spearman
83
+ value: 71.19680789937125
84
+ - type: manhattan_pearson
85
+ value: 69.57503405147493
86
+ - type: manhattan_spearman
87
+ value: 71.19417171802891
88
+ - task:
89
+ type: Clustering
90
+ dataset:
91
+ type: C-MTEB/CLSClusteringP2P
92
+ name: MTEB CLSClusteringP2P
93
+ config: default
94
+ split: test
95
+ revision: 4b6227591c6c1a73bc76b1055f3b7f3588e72476
96
+ metrics:
97
+ - type: v_measure
98
+ value: 40.006865414718185
99
+ - task:
100
+ type: Clustering
101
+ dataset:
102
+ type: C-MTEB/CLSClusteringS2S
103
+ name: MTEB CLSClusteringS2S
104
+ config: default
105
+ split: test
106
+ revision: e458b3f5414b62b7f9f83499ac1f5497ae2e869f
107
+ metrics:
108
+ - type: v_measure
109
+ value: 38.199450302104204
110
+ - task:
111
+ type: Reranking
112
+ dataset:
113
+ type: C-MTEB/CMedQAv1-reranking
114
+ name: MTEB CMedQAv1
115
+ config: default
116
+ split: test
117
+ revision: 8d7f1e942507dac42dc58017c1a001c3717da7df
118
+ metrics:
119
+ - type: map
120
+ value: 74.51766322144272
121
+ - type: mrr
122
+ value: 78.04591269841269
123
+ - task:
124
+ type: Reranking
125
+ dataset:
126
+ type: C-MTEB/CMedQAv2-reranking
127
+ name: MTEB CMedQAv2
128
+ config: default
129
+ split: test
130
+ revision: 23d186750531a14a0357ca22cd92d712fd512ea0
131
+ metrics:
132
+ - type: map
133
+ value: 75.6387269126998
134
+ - type: mrr
135
+ value: 79.5725
136
+ - task:
137
+ type: Retrieval
138
+ dataset:
139
+ type: C-MTEB/CmedqaRetrieval
140
+ name: MTEB CmedqaRetrieval
141
+ config: default
142
+ split: dev
143
+ revision: cd540c506dae1cf9e9a59c3e06f42030d54e7301
144
+ metrics:
145
+ - type: map_at_1
146
+ value: 16.788
147
+ - type: map_at_10
148
+ value: 25.224999999999998
149
+ - type: map_at_100
150
+ value: 26.862000000000002
151
+ - type: map_at_1000
152
+ value: 27.037
153
+ - type: map_at_3
154
+ value: 22.398
155
+ - type: map_at_5
156
+ value: 23.889
157
+ - type: mrr_at_1
158
+ value: 26.656999999999996
159
+ - type: mrr_at_10
160
+ value: 33.501999999999995
161
+ - type: mrr_at_100
162
+ value: 34.538999999999994
163
+ - type: mrr_at_1000
164
+ value: 34.626000000000005
165
+ - type: mrr_at_3
166
+ value: 31.374999999999996
167
+ - type: mrr_at_5
168
+ value: 32.535
169
+ - type: ndcg_at_1
170
+ value: 26.656999999999996
171
+ - type: ndcg_at_10
172
+ value: 30.675
173
+ - type: ndcg_at_100
174
+ value: 37.797
175
+ - type: ndcg_at_1000
176
+ value: 41.416
177
+ - type: ndcg_at_3
178
+ value: 26.827
179
+ - type: ndcg_at_5
180
+ value: 28.292
181
+ - type: precision_at_1
182
+ value: 26.656999999999996
183
+ - type: precision_at_10
184
+ value: 6.973999999999999
185
+ - type: precision_at_100
186
+ value: 1.286
187
+ - type: precision_at_1000
188
+ value: 0.174
189
+ - type: precision_at_3
190
+ value: 15.421000000000001
191
+ - type: precision_at_5
192
+ value: 11.133
193
+ - type: recall_at_1
194
+ value: 16.788
195
+ - type: recall_at_10
196
+ value: 38.746
197
+ - type: recall_at_100
198
+ value: 68.759
199
+ - type: recall_at_1000
200
+ value: 93.848
201
+ - type: recall_at_3
202
+ value: 26.807
203
+ - type: recall_at_5
204
+ value: 31.696999999999996
205
+ - task:
206
+ type: PairClassification
207
+ dataset:
208
+ type: C-MTEB/CMNLI
209
+ name: MTEB Cmnli
210
+ config: default
211
+ split: validation
212
+ revision: 41bc36f332156f7adc9e38f53777c959b2ae9766
213
+ metrics:
214
+ - type: cos_sim_accuracy
215
+ value: 61.14251352976549
216
+ - type: cos_sim_ap
217
+ value: 66.0067234682187
218
+ - type: cos_sim_f1
219
+ value: 68.08438532576463
220
+ - type: cos_sim_precision
221
+ value: 53.38825405261759
222
+ - type: cos_sim_recall
223
+ value: 93.9443535188216
224
+ - type: dot_accuracy
225
+ value: 61.14251352976549
226
+ - type: dot_ap
227
+ value: 65.99885462184204
228
+ - type: dot_f1
229
+ value: 68.08438532576463
230
+ - type: dot_precision
231
+ value: 53.38825405261759
232
+ - type: dot_recall
233
+ value: 93.9443535188216
234
+ - type: euclidean_accuracy
235
+ value: 61.14251352976549
236
+ - type: euclidean_ap
237
+ value: 66.0068124858108
238
+ - type: euclidean_f1
239
+ value: 68.08438532576463
240
+ - type: euclidean_precision
241
+ value: 53.38825405261759
242
+ - type: euclidean_recall
243
+ value: 93.9443535188216
244
+ - type: manhattan_accuracy
245
+ value: 61.10643415514131
246
+ - type: manhattan_ap
247
+ value: 66.00394480408306
248
+ - type: manhattan_f1
249
+ value: 68.1290759718811
250
+ - type: manhattan_precision
251
+ value: 53.41301460823373
252
+ - type: manhattan_recall
253
+ value: 94.03787701660042
254
+ - type: max_accuracy
255
+ value: 61.14251352976549
256
+ - type: max_ap
257
+ value: 66.0068124858108
258
+ - type: max_f1
259
+ value: 68.1290759718811
260
+ - task:
261
+ type: Retrieval
262
+ dataset:
263
+ type: C-MTEB/CovidRetrieval
264
+ name: MTEB CovidRetrieval
265
+ config: default
266
+ split: dev
267
+ revision: 1271c7809071a13532e05f25fb53511ffce77117
268
+ metrics:
269
+ - type: map_at_1
270
+ value: 49.315
271
+ - type: map_at_10
272
+ value: 59.998
273
+ - type: map_at_100
274
+ value: 60.649
275
+ - type: map_at_1000
276
+ value: 60.668
277
+ - type: map_at_3
278
+ value: 57.727
279
+ - type: map_at_5
280
+ value: 59.019999999999996
281
+ - type: mrr_at_1
282
+ value: 49.419999999999995
283
+ - type: mrr_at_10
284
+ value: 60.036
285
+ - type: mrr_at_100
286
+ value: 60.678
287
+ - type: mrr_at_1000
288
+ value: 60.697
289
+ - type: mrr_at_3
290
+ value: 57.833
291
+ - type: mrr_at_5
292
+ value: 59.097
293
+ - type: ndcg_at_1
294
+ value: 49.419999999999995
295
+ - type: ndcg_at_10
296
+ value: 64.976
297
+ - type: ndcg_at_100
298
+ value: 68.068
299
+ - type: ndcg_at_1000
300
+ value: 68.587
301
+ - type: ndcg_at_3
302
+ value: 60.358000000000004
303
+ - type: ndcg_at_5
304
+ value: 62.668
305
+ - type: precision_at_1
306
+ value: 49.419999999999995
307
+ - type: precision_at_10
308
+ value: 8.124
309
+ - type: precision_at_100
310
+ value: 0.958
311
+ - type: precision_at_1000
312
+ value: 0.1
313
+ - type: precision_at_3
314
+ value: 22.726
315
+ - type: precision_at_5
316
+ value: 14.795
317
+ - type: recall_at_1
318
+ value: 49.315
319
+ - type: recall_at_10
320
+ value: 80.453
321
+ - type: recall_at_100
322
+ value: 94.731
323
+ - type: recall_at_1000
324
+ value: 98.84100000000001
325
+ - type: recall_at_3
326
+ value: 67.861
327
+ - type: recall_at_5
328
+ value: 73.393
329
+ - task:
330
+ type: Retrieval
331
+ dataset:
332
+ type: C-MTEB/DuRetrieval
333
+ name: MTEB DuRetrieval
334
+ config: default
335
+ split: dev
336
+ revision: a1a333e290fe30b10f3f56498e3a0d911a693ced
337
+ metrics:
338
+ - type: map_at_1
339
+ value: 21.493000000000002
340
+ - type: map_at_10
341
+ value: 64.066
342
+ - type: map_at_100
343
+ value: 67.917
344
+ - type: map_at_1000
345
+ value: 68.02
346
+ - type: map_at_3
347
+ value: 43.847
348
+ - type: map_at_5
349
+ value: 55.18300000000001
350
+ - type: mrr_at_1
351
+ value: 76.5
352
+ - type: mrr_at_10
353
+ value: 84.057
354
+ - type: mrr_at_100
355
+ value: 84.19
356
+ - type: mrr_at_1000
357
+ value: 84.195
358
+ - type: mrr_at_3
359
+ value: 83.125
360
+ - type: mrr_at_5
361
+ value: 83.817
362
+ - type: ndcg_at_1
363
+ value: 76.5
364
+ - type: ndcg_at_10
365
+ value: 74.601
366
+ - type: ndcg_at_100
367
+ value: 80.072
368
+ - type: ndcg_at_1000
369
+ value: 81.047
370
+ - type: ndcg_at_3
371
+ value: 72.16799999999999
372
+ - type: ndcg_at_5
373
+ value: 71.479
374
+ - type: precision_at_1
375
+ value: 76.5
376
+ - type: precision_at_10
377
+ value: 36.385
378
+ - type: precision_at_100
379
+ value: 4.633
380
+ - type: precision_at_1000
381
+ value: 0.48700000000000004
382
+ - type: precision_at_3
383
+ value: 64.617
384
+ - type: precision_at_5
385
+ value: 54.89000000000001
386
+ - type: recall_at_1
387
+ value: 21.493000000000002
388
+ - type: recall_at_10
389
+ value: 77.387
390
+ - type: recall_at_100
391
+ value: 93.996
392
+ - type: recall_at_1000
393
+ value: 98.83
394
+ - type: recall_at_3
395
+ value: 47.563
396
+ - type: recall_at_5
397
+ value: 62.883
398
+ - task:
399
+ type: Retrieval
400
+ dataset:
401
+ type: C-MTEB/EcomRetrieval
402
+ name: MTEB EcomRetrieval
403
+ config: default
404
+ split: dev
405
+ revision: 687de13dc7294d6fd9be10c6945f9e8fec8166b9
406
+ metrics:
407
+ - type: map_at_1
408
+ value: 28.499999999999996
409
+ - type: map_at_10
410
+ value: 37.271
411
+ - type: map_at_100
412
+ value: 38.318000000000005
413
+ - type: map_at_1000
414
+ value: 38.362
415
+ - type: map_at_3
416
+ value: 34.2
417
+ - type: map_at_5
418
+ value: 35.97
419
+ - type: mrr_at_1
420
+ value: 28.499999999999996
421
+ - type: mrr_at_10
422
+ value: 37.271
423
+ - type: mrr_at_100
424
+ value: 38.318000000000005
425
+ - type: mrr_at_1000
426
+ value: 38.362
427
+ - type: mrr_at_3
428
+ value: 34.2
429
+ - type: mrr_at_5
430
+ value: 35.97
431
+ - type: ndcg_at_1
432
+ value: 28.499999999999996
433
+ - type: ndcg_at_10
434
+ value: 42.419000000000004
435
+ - type: ndcg_at_100
436
+ value: 47.591
437
+ - type: ndcg_at_1000
438
+ value: 48.791000000000004
439
+ - type: ndcg_at_3
440
+ value: 36.074
441
+ - type: ndcg_at_5
442
+ value: 39.275
443
+ - type: precision_at_1
444
+ value: 28.499999999999996
445
+ - type: precision_at_10
446
+ value: 5.8999999999999995
447
+ - type: precision_at_100
448
+ value: 0.8340000000000001
449
+ - type: precision_at_1000
450
+ value: 0.093
451
+ - type: precision_at_3
452
+ value: 13.833
453
+ - type: precision_at_5
454
+ value: 9.86
455
+ - type: recall_at_1
456
+ value: 28.499999999999996
457
+ - type: recall_at_10
458
+ value: 59.0
459
+ - type: recall_at_100
460
+ value: 83.39999999999999
461
+ - type: recall_at_1000
462
+ value: 92.9
463
+ - type: recall_at_3
464
+ value: 41.5
465
+ - type: recall_at_5
466
+ value: 49.3
467
+ - task:
468
+ type: Classification
469
+ dataset:
470
+ type: C-MTEB/IFlyTek-classification
471
+ name: MTEB IFlyTek
472
+ config: default
473
+ split: validation
474
+ revision: 421605374b29664c5fc098418fe20ada9bd55f8a
475
+ metrics:
476
+ - type: accuracy
477
+ value: 44.563293574451706
478
+ - type: f1
479
+ value: 30.778378421912002
480
+ - task:
481
+ type: Classification
482
+ dataset:
483
+ type: C-MTEB/JDReview-classification
484
+ name: MTEB JDReview
485
+ config: default
486
+ split: test
487
+ revision: b7c64bd89eb87f8ded463478346f76731f07bf8b
488
+ metrics:
489
+ - type: accuracy
490
+ value: 89.23076923076924
491
+ - type: ap
492
+ value: 61.032723799932974
493
+ - type: f1
494
+ value: 84.49441280498773
495
+ - task:
496
+ type: STS
497
+ dataset:
498
+ type: C-MTEB/LCQMC
499
+ name: MTEB LCQMC
500
+ config: default
501
+ split: test
502
+ revision: 17f9b096f80380fce5ed12a9be8be7784b337daf
503
+ metrics:
504
+ - type: cos_sim_pearson
505
+ value: 66.22719629640424
506
+ - type: cos_sim_spearman
507
+ value: 72.87558846789628
508
+ - type: euclidean_pearson
509
+ value: 71.17001015918034
510
+ - type: euclidean_spearman
511
+ value: 72.8755973307354
512
+ - type: manhattan_pearson
513
+ value: 71.27354402109685
514
+ - type: manhattan_spearman
515
+ value: 72.9741374432873
516
+ - task:
517
+ type: Reranking
518
+ dataset:
519
+ type: C-MTEB/Mmarco-reranking
520
+ name: MTEB MMarcoReranking
521
+ config: default
522
+ split: dev
523
+ revision: 8e0c766dbe9e16e1d221116a3f36795fbade07f6
524
+ metrics:
525
+ - type: map
526
+ value: 17.12412338289922
527
+ - type: mrr
528
+ value: 15.632936507936506
529
+ - task:
530
+ type: Retrieval
531
+ dataset:
532
+ type: C-MTEB/MMarcoRetrieval
533
+ name: MTEB MMarcoRetrieval
534
+ config: default
535
+ split: dev
536
+ revision: 539bbde593d947e2a124ba72651aafc09eb33fc2
537
+ metrics:
538
+ - type: map_at_1
539
+ value: 43.888
540
+ - type: map_at_10
541
+ value: 53.053
542
+ - type: map_at_100
543
+ value: 53.702000000000005
544
+ - type: map_at_1000
545
+ value: 53.739000000000004
546
+ - type: map_at_3
547
+ value: 50.613
548
+ - type: map_at_5
549
+ value: 52.035
550
+ - type: mrr_at_1
551
+ value: 45.587
552
+ - type: mrr_at_10
553
+ value: 53.921
554
+ - type: mrr_at_100
555
+ value: 54.51200000000001
556
+ - type: mrr_at_1000
557
+ value: 54.54599999999999
558
+ - type: mrr_at_3
559
+ value: 51.690999999999995
560
+ - type: mrr_at_5
561
+ value: 52.977
562
+ - type: ndcg_at_1
563
+ value: 45.587
564
+ - type: ndcg_at_10
565
+ value: 57.619
566
+ - type: ndcg_at_100
567
+ value: 60.76200000000001
568
+ - type: ndcg_at_1000
569
+ value: 61.797000000000004
570
+ - type: ndcg_at_3
571
+ value: 52.805
572
+ - type: ndcg_at_5
573
+ value: 55.239000000000004
574
+ - type: precision_at_1
575
+ value: 45.587
576
+ - type: precision_at_10
577
+ value: 7.564
578
+ - type: precision_at_100
579
+ value: 0.915
580
+ - type: precision_at_1000
581
+ value: 0.1
582
+ - type: precision_at_3
583
+ value: 20.468
584
+ - type: precision_at_5
585
+ value: 13.572999999999999
586
+ - type: recall_at_1
587
+ value: 43.888
588
+ - type: recall_at_10
589
+ value: 71.06700000000001
590
+ - type: recall_at_100
591
+ value: 85.765
592
+ - type: recall_at_1000
593
+ value: 94.038
594
+ - type: recall_at_3
595
+ value: 58.069
596
+ - type: recall_at_5
597
+ value: 63.848000000000006
598
+ - task:
599
+ type: Classification
600
+ dataset:
601
+ type: mteb/amazon_massive_intent
602
+ name: MTEB MassiveIntentClassification (zh-CN)
603
+ config: zh-CN
604
+ split: test
605
+ revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
606
+ metrics:
607
+ - type: accuracy
608
+ value: 60.423671822461344
609
+ - type: f1
610
+ value: 56.82053357104769
611
+ - task:
612
+ type: Classification
613
+ dataset:
614
+ type: mteb/amazon_massive_scenario
615
+ name: MTEB MassiveScenarioClassification (zh-CN)
616
+ config: zh-CN
617
+ split: test
618
+ revision: 7d571f92784cd94a019292a1f45445077d0ef634
619
+ metrics:
620
+ - type: accuracy
621
+ value: 68.69199731002018
622
+ - type: f1
623
+ value: 68.36036256101542
624
+ - task:
625
+ type: Retrieval
626
+ dataset:
627
+ type: C-MTEB/MedicalRetrieval
628
+ name: MTEB MedicalRetrieval
629
+ config: default
630
+ split: dev
631
+ revision: 2039188fb5800a9803ba5048df7b76e6fb151fc6
632
+ metrics:
633
+ - type: map_at_1
634
+ value: 38.0
635
+ - type: map_at_10
636
+ value: 43.6
637
+ - type: map_at_100
638
+ value: 44.235
639
+ - type: map_at_1000
640
+ value: 44.299
641
+ - type: map_at_3
642
+ value: 42.016999999999996
643
+ - type: map_at_5
644
+ value: 42.862
645
+ - type: mrr_at_1
646
+ value: 38.1
647
+ - type: mrr_at_10
648
+ value: 43.65
649
+ - type: mrr_at_100
650
+ value: 44.284
651
+ - type: mrr_at_1000
652
+ value: 44.348
653
+ - type: mrr_at_3
654
+ value: 42.067
655
+ - type: mrr_at_5
656
+ value: 42.912
657
+ - type: ndcg_at_1
658
+ value: 38.0
659
+ - type: ndcg_at_10
660
+ value: 46.537
661
+ - type: ndcg_at_100
662
+ value: 49.936
663
+ - type: ndcg_at_1000
664
+ value: 51.925
665
+ - type: ndcg_at_3
666
+ value: 43.251
667
+ - type: ndcg_at_5
668
+ value: 44.753
669
+ - type: precision_at_1
670
+ value: 38.0
671
+ - type: precision_at_10
672
+ value: 5.59
673
+ - type: precision_at_100
674
+ value: 0.7250000000000001
675
+ - type: precision_at_1000
676
+ value: 0.089
677
+ - type: precision_at_3
678
+ value: 15.6
679
+ - type: precision_at_5
680
+ value: 10.08
681
+ - type: recall_at_1
682
+ value: 38.0
683
+ - type: recall_at_10
684
+ value: 55.900000000000006
685
+ - type: recall_at_100
686
+ value: 72.5
687
+ - type: recall_at_1000
688
+ value: 88.8
689
+ - type: recall_at_3
690
+ value: 46.800000000000004
691
+ - type: recall_at_5
692
+ value: 50.4
693
+ - task:
694
+ type: Retrieval
695
+ dataset:
696
+ type: Shitao/MLDR
697
+ name: MTEB MultiLongDocRetrieval (zh)
698
+ config: zh
699
+ split: test
700
+ revision: None
701
+ metrics:
702
+ - type: map_at_1
703
+ value: 7.75
704
+ - type: map_at_10
705
+ value: 10.508000000000001
706
+ - type: map_at_100
707
+ value: 10.988000000000001
708
+ - type: map_at_1000
709
+ value: 11.059
710
+ - type: map_at_3
711
+ value: 9.417
712
+ - type: map_at_5
713
+ value: 9.942
714
+ - type: mrr_at_1
715
+ value: 7.75
716
+ - type: mrr_at_10
717
+ value: 10.508000000000001
718
+ - type: mrr_at_100
719
+ value: 10.988000000000001
720
+ - type: mrr_at_1000
721
+ value: 11.06
722
+ - type: mrr_at_3
723
+ value: 9.417
724
+ - type: mrr_at_5
725
+ value: 9.942
726
+ - type: ndcg_at_1
727
+ value: 7.75
728
+ - type: ndcg_at_10
729
+ value: 12.315
730
+ - type: ndcg_at_100
731
+ value: 15.018999999999998
732
+ - type: ndcg_at_1000
733
+ value: 17.424999999999997
734
+ - type: ndcg_at_3
735
+ value: 9.982000000000001
736
+ - type: ndcg_at_5
737
+ value: 10.918
738
+ - type: precision_at_1
739
+ value: 7.75
740
+ - type: precision_at_10
741
+ value: 1.825
742
+ - type: precision_at_100
743
+ value: 0.317
744
+ - type: precision_at_1000
745
+ value: 0.052
746
+ - type: precision_at_3
747
+ value: 3.875
748
+ - type: precision_at_5
749
+ value: 2.775
750
+ - type: recall_at_1
751
+ value: 7.75
752
+ - type: recall_at_10
753
+ value: 18.25
754
+ - type: recall_at_100
755
+ value: 31.75
756
+ - type: recall_at_1000
757
+ value: 51.74999999999999
758
+ - type: recall_at_3
759
+ value: 11.625
760
+ - type: recall_at_5
761
+ value: 13.875000000000002
762
+ - task:
763
+ type: Classification
764
+ dataset:
765
+ type: C-MTEB/MultilingualSentiment-classification
766
+ name: MTEB MultilingualSentiment
767
+ config: default
768
+ split: validation
769
+ revision: 46958b007a63fdbf239b7672c25d0bea67b5ea1a
770
+ metrics:
771
+ - type: accuracy
772
+ value: 78.42
773
+ - type: f1
774
+ value: 78.06136479834592
775
+ - task:
776
+ type: PairClassification
777
+ dataset:
778
+ type: C-MTEB/OCNLI
779
+ name: MTEB Ocnli
780
+ config: default
781
+ split: validation
782
+ revision: 66e76a618a34d6d565d5538088562851e6daa7ec
783
+ metrics:
784
+ - type: cos_sim_accuracy
785
+ value: 59.61017866811045
786
+ - type: cos_sim_ap
787
+ value: 60.271467647520495
788
+ - type: cos_sim_f1
789
+ value: 69.19575113808801
790
+ - type: cos_sim_precision
791
+ value: 53.99644760213144
792
+ - type: cos_sim_recall
793
+ value: 96.30411826821542
794
+ - type: dot_accuracy
795
+ value: 59.61017866811045
796
+ - type: dot_ap
797
+ value: 60.271467647520495
798
+ - type: dot_f1
799
+ value: 69.19575113808801
800
+ - type: dot_precision
801
+ value: 53.99644760213144
802
+ - type: dot_recall
803
+ value: 96.30411826821542
804
+ - type: euclidean_accuracy
805
+ value: 59.61017866811045
806
+ - type: euclidean_ap
807
+ value: 60.271467647520495
808
+ - type: euclidean_f1
809
+ value: 69.19575113808801
810
+ - type: euclidean_precision
811
+ value: 53.99644760213144
812
+ - type: euclidean_recall
813
+ value: 96.30411826821542
814
+ - type: manhattan_accuracy
815
+ value: 59.7184623714131
816
+ - type: manhattan_ap
817
+ value: 60.32264902752218
818
+ - type: manhattan_f1
819
+ value: 69.22201138519924
820
+ - type: manhattan_precision
821
+ value: 54.02843601895735
822
+ - type: manhattan_recall
823
+ value: 96.30411826821542
824
+ - type: max_accuracy
825
+ value: 59.7184623714131
826
+ - type: max_ap
827
+ value: 60.32264902752218
828
+ - type: max_f1
829
+ value: 69.22201138519924
830
+ - task:
831
+ type: Classification
832
+ dataset:
833
+ type: C-MTEB/OnlineShopping-classification
834
+ name: MTEB OnlineShopping
835
+ config: default
836
+ split: test
837
+ revision: e610f2ebd179a8fda30ae534c3878750a96db120
838
+ metrics:
839
+ - type: accuracy
840
+ value: 93.05000000000001
841
+ - type: ap
842
+ value: 91.18195895507802
843
+ - type: f1
844
+ value: 93.04021920382944
845
+ - task:
846
+ type: STS
847
+ dataset:
848
+ type: C-MTEB/PAWSX
849
+ name: MTEB PAWSX
850
+ config: default
851
+ split: test
852
+ revision: 9c6a90e430ac22b5779fb019a23e820b11a8b5e1
853
+ metrics:
854
+ - type: cos_sim_pearson
855
+ value: 15.646669268679258
856
+ - type: cos_sim_spearman
857
+ value: 18.210627988600088
858
+ - type: euclidean_pearson
859
+ value: 18.545700775834135
860
+ - type: euclidean_spearman
861
+ value: 18.22638450822432
862
+ - type: manhattan_pearson
863
+ value: 18.56718129015248
864
+ - type: manhattan_spearman
865
+ value: 18.27028021184377
866
+ - task:
867
+ type: PairClassification
868
+ dataset:
869
+ type: paws-x
870
+ name: MTEB PawsX (zh)
871
+ config: zh
872
+ split: test
873
+ revision: 8a04d940a42cd40658986fdd8e3da561533a3646
874
+ metrics:
875
+ - type: cos_sim_accuracy
876
+ value: 59.599999999999994
877
+ - type: cos_sim_ap
878
+ value: 57.39257507549949
879
+ - type: cos_sim_f1
880
+ value: 62.32136632973162
881
+ - type: cos_sim_precision
882
+ value: 45.265822784810126
883
+ - type: cos_sim_recall
884
+ value: 100.0
885
+ - type: dot_accuracy
886
+ value: 59.599999999999994
887
+ - type: dot_ap
888
+ value: 56.98345226457827
889
+ - type: dot_f1
890
+ value: 62.32136632973162
891
+ - type: dot_precision
892
+ value: 45.265822784810126
893
+ - type: dot_recall
894
+ value: 100.0
895
+ - type: euclidean_accuracy
896
+ value: 59.599999999999994
897
+ - type: euclidean_ap
898
+ value: 57.3922995193463
899
+ - type: euclidean_f1
900
+ value: 62.32136632973162
901
+ - type: euclidean_precision
902
+ value: 45.265822784810126
903
+ - type: euclidean_recall
904
+ value: 100.0
905
+ - type: manhattan_accuracy
906
+ value: 59.550000000000004
907
+ - type: manhattan_ap
908
+ value: 57.409507119268376
909
+ - type: manhattan_f1
910
+ value: 62.32136632973162
911
+ - type: manhattan_precision
912
+ value: 45.265822784810126
913
+ - type: manhattan_recall
914
+ value: 100.0
915
+ - type: max_accuracy
916
+ value: 59.599999999999994
917
+ - type: max_ap
918
+ value: 57.409507119268376
919
+ - type: max_f1
920
+ value: 62.32136632973162
921
+ - task:
922
+ type: STS
923
+ dataset:
924
+ type: C-MTEB/QBQTC
925
+ name: MTEB QBQTC
926
+ config: default
927
+ split: test
928
+ revision: 790b0510dc52b1553e8c49f3d2afb48c0e5c48b7
929
+ metrics:
930
+ - type: cos_sim_pearson
931
+ value: 26.20813788138459
932
+ - type: cos_sim_spearman
933
+ value: 27.161390755734427
934
+ - type: euclidean_pearson
935
+ value: 25.38239308132396
936
+ - type: euclidean_spearman
937
+ value: 27.161458517606906
938
+ - type: manhattan_pearson
939
+ value: 25.533861242377704
940
+ - type: manhattan_spearman
941
+ value: 27.356882292991834
942
+ - task:
943
+ type: STS
944
+ dataset:
945
+ type: mteb/sts22-crosslingual-sts
946
+ name: MTEB STS22 (zh)
947
+ config: zh
948
+ split: test
949
+ revision: eea2b4fe26a775864c896887d910b76a8098ad3f
950
+ metrics:
951
+ - type: cos_sim_pearson
952
+ value: 63.437181390975624
953
+ - type: cos_sim_spearman
954
+ value: 66.42410464140018
955
+ - type: euclidean_pearson
956
+ value: 65.23091221659769
957
+ - type: euclidean_spearman
958
+ value: 66.42410464140018
959
+ - type: manhattan_pearson
960
+ value: 65.42430798528258
961
+ - type: manhattan_spearman
962
+ value: 66.44811980407968
963
+ - task:
964
+ type: STS
965
+ dataset:
966
+ type: C-MTEB/STSB
967
+ name: MTEB STSB
968
+ config: default
969
+ split: test
970
+ revision: 0cde68302b3541bb8b3c340dc0644b0b745b3dc0
971
+ metrics:
972
+ - type: cos_sim_pearson
973
+ value: 67.13526185743584
974
+ - type: cos_sim_spearman
975
+ value: 67.87262977922003
976
+ - type: euclidean_pearson
977
+ value: 68.01501802788067
978
+ - type: euclidean_spearman
979
+ value: 67.87241377586508
980
+ - type: manhattan_pearson
981
+ value: 68.03194534033594
982
+ - type: manhattan_spearman
983
+ value: 67.91448799292998
984
+ - task:
985
+ type: Reranking
986
+ dataset:
987
+ type: C-MTEB/T2Reranking
988
+ name: MTEB T2Reranking
989
+ config: default
990
+ split: dev
991
+ revision: 76631901a18387f85eaa53e5450019b87ad58ef9
992
+ metrics:
993
+ - type: map
994
+ value: 64.41164023353707
995
+ - type: mrr
996
+ value: 74.15855283661647
997
+ - task:
998
+ type: Retrieval
999
+ dataset:
1000
+ type: C-MTEB/T2Retrieval
1001
+ name: MTEB T2Retrieval
1002
+ config: default
1003
+ split: dev
1004
+ revision: 8731a845f1bf500a4f111cf1070785c793d10e64
1005
+ metrics:
1006
+ - type: map_at_1
1007
+ value: 20.456
1008
+ - type: map_at_10
1009
+ value: 57.611999999999995
1010
+ - type: map_at_100
1011
+ value: 62.104000000000006
1012
+ - type: map_at_1000
1013
+ value: 62.251999999999995
1014
+ - type: map_at_3
1015
+ value: 39.806999999999995
1016
+ - type: map_at_5
1017
+ value: 49.016999999999996
1018
+ - type: mrr_at_1
1019
+ value: 73.685
1020
+ - type: mrr_at_10
1021
+ value: 79.361
1022
+ - type: mrr_at_100
1023
+ value: 79.63799999999999
1024
+ - type: mrr_at_1000
1025
+ value: 79.649
1026
+ - type: mrr_at_3
1027
+ value: 78.144
1028
+ - type: mrr_at_5
1029
+ value: 78.89699999999999
1030
+ - type: ndcg_at_1
1031
+ value: 73.685
1032
+ - type: ndcg_at_10
1033
+ value: 67.824
1034
+ - type: ndcg_at_100
1035
+ value: 74.399
1036
+ - type: ndcg_at_1000
1037
+ value: 75.949
1038
+ - type: ndcg_at_3
1039
+ value: 68.643
1040
+ - type: ndcg_at_5
1041
+ value: 67.108
1042
+ - type: precision_at_1
1043
+ value: 73.685
1044
+ - type: precision_at_10
1045
+ value: 34.904
1046
+ - type: precision_at_100
1047
+ value: 4.714
1048
+ - type: precision_at_1000
1049
+ value: 0.508
1050
+ - type: precision_at_3
1051
+ value: 60.587
1052
+ - type: precision_at_5
1053
+ value: 50.892
1054
+ - type: recall_at_1
1055
+ value: 20.456
1056
+ - type: recall_at_10
1057
+ value: 68.314
1058
+ - type: recall_at_100
1059
+ value: 88.67399999999999
1060
+ - type: recall_at_1000
1061
+ value: 96.48400000000001
1062
+ - type: recall_at_3
1063
+ value: 42.498999999999995
1064
+ - type: recall_at_5
1065
+ value: 54.492
1066
+ - task:
1067
+ type: Classification
1068
+ dataset:
1069
+ type: C-MTEB/TNews-classification
1070
+ name: MTEB TNews
1071
+ config: default
1072
+ split: validation
1073
+ revision: 317f262bf1e6126357bbe89e875451e4b0938fe4
1074
+ metrics:
1075
+ - type: accuracy
1076
+ value: 50.258
1077
+ - type: f1
1078
+ value: 48.46817969810712
1079
+ - task:
1080
+ type: Clustering
1081
+ dataset:
1082
+ type: C-MTEB/ThuNewsClusteringP2P
1083
+ name: MTEB ThuNewsClusteringP2P
1084
+ config: default
1085
+ split: test
1086
+ revision: 5798586b105c0434e4f0fe5e767abe619442cf93
1087
+ metrics:
1088
+ - type: v_measure
1089
+ value: 59.39920752654844
1090
+ - task:
1091
+ type: Clustering
1092
+ dataset:
1093
+ type: C-MTEB/ThuNewsClusteringS2S
1094
+ name: MTEB ThuNewsClusteringS2S
1095
+ config: default
1096
+ split: test
1097
+ revision: 8a8b2caeda43f39e13c4bc5bea0f8a667896e10d
1098
+ metrics:
1099
+ - type: v_measure
1100
+ value: 55.52800428947542
1101
+ - task:
1102
+ type: Retrieval
1103
+ dataset:
1104
+ type: C-MTEB/VideoRetrieval
1105
+ name: MTEB VideoRetrieval
1106
+ config: default
1107
+ split: dev
1108
+ revision: 58c2597a5943a2ba48f4668c3b90d796283c5639
1109
+ metrics:
1110
+ - type: map_at_1
1111
+ value: 25.4
1112
+ - type: map_at_10
1113
+ value: 33.43
1114
+ - type: map_at_100
1115
+ value: 34.259
1116
+ - type: map_at_1000
1117
+ value: 34.329
1118
+ - type: map_at_3
1119
+ value: 30.817
1120
+ - type: map_at_5
1121
+ value: 32.422000000000004
1122
+ - type: mrr_at_1
1123
+ value: 25.3
1124
+ - type: mrr_at_10
1125
+ value: 33.379999999999995
1126
+ - type: mrr_at_100
1127
+ value: 34.209
1128
+ - type: mrr_at_1000
1129
+ value: 34.278999999999996
1130
+ - type: mrr_at_3
1131
+ value: 30.767
1132
+ - type: mrr_at_5
1133
+ value: 32.372
1134
+ - type: ndcg_at_1
1135
+ value: 25.4
1136
+ - type: ndcg_at_10
1137
+ value: 37.797
1138
+ - type: ndcg_at_100
1139
+ value: 42.168
1140
+ - type: ndcg_at_1000
1141
+ value: 44.194
1142
+ - type: ndcg_at_3
1143
+ value: 32.537
1144
+ - type: ndcg_at_5
1145
+ value: 35.403
1146
+ - type: precision_at_1
1147
+ value: 25.4
1148
+ - type: precision_at_10
1149
+ value: 5.17
1150
+ - type: precision_at_100
1151
+ value: 0.73
1152
+ - type: precision_at_1000
1153
+ value: 0.089
1154
+ - type: precision_at_3
1155
+ value: 12.5
1156
+ - type: precision_at_5
1157
+ value: 8.88
1158
+ - type: recall_at_1
1159
+ value: 25.4
1160
+ - type: recall_at_10
1161
+ value: 51.7
1162
+ - type: recall_at_100
1163
+ value: 73.0
1164
+ - type: recall_at_1000
1165
+ value: 89.3
1166
+ - type: recall_at_3
1167
+ value: 37.5
1168
+ - type: recall_at_5
1169
+ value: 44.4
1170
+ - task:
1171
+ type: Classification
1172
+ dataset:
1173
+ type: C-MTEB/waimai-classification
1174
+ name: MTEB Waimai
1175
+ config: default
1176
+ split: test
1177
+ revision: 339287def212450dcaa9df8c22bf93e9980c7023
1178
+ metrics:
1179
+ - type: accuracy
1180
+ value: 89.17000000000002
1181
+ - type: ap
1182
+ value: 74.83484198968617
1183
+ - type: f1
1184
+ value: 87.84607916808504
1185
  ---