INFO: 2024-11-15 16:53:45,193: llmtf.base.evaluator: Starting eval on ['darumeru/multiq'] INFO: 2024-11-15 16:53:45,193: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 16:53:45,193: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 16:53:45,368: llmtf.base.evaluator: Starting eval on ['darumeru/parus'] INFO: 2024-11-15 16:53:45,368: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 16:53:45,368: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 16:53:47,651: llmtf.base.darumeru/PARus: Loading Dataset: 2.28s INFO: 2024-11-15 16:53:48,903: llmtf.base.darumeru/MultiQ: Loading Dataset: 3.71s INFO: 2024-11-15 16:53:49,849: llmtf.base.darumeru/PARus: Processing Dataset: 2.20s INFO: 2024-11-15 16:53:49,849: llmtf.base.darumeru/PARus: Results for darumeru/PARus: INFO: 2024-11-15 16:53:49,859: llmtf.base.darumeru/PARus: {'acc': 0.21} INFO: 2024-11-15 16:53:49,859: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 16:53:49,859: llmtf.base.evaluator: mean darumeru/PARus 0.210 0.210 INFO: 2024-11-15 16:53:58,225: llmtf.base.evaluator: Starting eval on ['darumeru/ruopenbookqa'] INFO: 2024-11-15 16:53:58,225: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 16:53:58,225: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 16:54:01,621: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 3.40s INFO: 2024-11-15 16:54:27,891: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 26.27s INFO: 2024-11-15 16:54:27,892: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: INFO: 2024-11-15 16:54:27,903: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.46134020618556704, 'f1_macro': 0.4607981854644543} INFO: 2024-11-15 16:54:27,909: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 16:54:27,910: llmtf.base.evaluator: mean darumeru/PARus darumeru/ruOpenBookQA 0.336 0.210 0.461 INFO: 2024-11-15 16:54:36,808: llmtf.base.evaluator: Starting eval on ['darumeru/rwsd'] INFO: 2024-11-15 16:54:36,809: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 16:54:36,809: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 16:54:41,755: llmtf.base.darumeru/RWSD: Loading Dataset: 4.95s INFO: 2024-11-15 16:54:44,248: llmtf.base.darumeru/RWSD: Processing Dataset: 2.49s INFO: 2024-11-15 16:54:44,248: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: INFO: 2024-11-15 16:54:44,248: llmtf.base.darumeru/RWSD: {'acc': 0.45098039215686275} INFO: 2024-11-15 16:54:44,249: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 16:54:44,250: llmtf.base.evaluator: mean darumeru/PARus darumeru/RWSD darumeru/ruOpenBookQA 0.374 0.210 0.451 0.461 INFO: 2024-11-15 16:54:52,912: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] INFO: 2024-11-15 16:54:52,912: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 16:54:52,912: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 16:56:43,744: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 110.83s INFO: 2024-11-15 16:59:38,517: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 174.77s INFO: 2024-11-15 16:59:38,517: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: INFO: 2024-11-15 16:59:38,578: llmtf.base.nlpcoreteam/ruMMLU: metric subject abstract_algebra 0.240000 anatomy 0.370370 astronomy 0.453947 business_ethics 0.350000 clinical_knowledge 0.475472 college_biology 0.340278 college_chemistry 0.250000 college_computer_science 0.350000 college_mathematics 0.310000 college_medicine 0.427746 college_physics 0.274510 computer_security 0.540000 conceptual_physics 0.331915 econometrics 0.350877 electrical_engineering 0.496552 elementary_mathematics 0.357143 formal_logic 0.317460 global_facts 0.350000 high_school_biology 0.416129 high_school_chemistry 0.438424 high_school_computer_science 0.550000 high_school_european_history 0.472727 high_school_geography 0.484848 high_school_government_and_politics 0.398964 high_school_macroeconomics 0.325641 high_school_mathematics 0.318519 high_school_microeconomics 0.407563 high_school_physics 0.337748 high_school_psychology 0.453211 high_school_statistics 0.342593 high_school_us_history 0.387255 high_school_world_history 0.468354 human_aging 0.448430 human_sexuality 0.396947 international_law 0.661157 jurisprudence 0.490741 logical_fallacies 0.312883 machine_learning 0.339286 management 0.446602 marketing 0.615385 medical_genetics 0.390000 miscellaneous 0.441890 moral_disputes 0.419075 moral_scenarios 0.242458 nutrition 0.411765 philosophy 0.456592 prehistory 0.373457 professional_accounting 0.336879 professional_law 0.314211 professional_medicine 0.330882 professional_psychology 0.364379 public_relations 0.463636 security_studies 0.395918 sociology 0.452736 us_foreign_policy 0.640000 virology 0.361446 world_religions 0.403509 INFO: 2024-11-15 16:59:38,586: llmtf.base.nlpcoreteam/ruMMLU: metric subject STEM 0.371502 humanities 0.409222 other (business, health, misc.) 0.411205 social sciences 0.427893 INFO: 2024-11-15 16:59:38,591: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.4049555256540374} INFO: 2024-11-15 16:59:38,625: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 16:59:38,627: llmtf.base.evaluator: mean darumeru/PARus darumeru/RWSD darumeru/ruOpenBookQA nlpcoreteam/ruMMLU 0.382 0.210 0.451 0.461 0.405 INFO: 2024-11-15 16:59:47,415: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] INFO: 2024-11-15 16:59:47,416: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 16:59:47,416: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 16:59:50,756: llmtf.base.daru/treewayabstractive: Loading Dataset: 3.34s INFO: 2024-11-15 17:00:04,037: llmtf.base.darumeru/MultiQ: Processing Dataset: 375.13s INFO: 2024-11-15 17:00:04,037: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: INFO: 2024-11-15 17:00:04,038: llmtf.base.darumeru/MultiQ: {'f1': 0.2166827508040571, 'em': 0.11567877629063097} INFO: 2024-11-15 17:00:04,043: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 17:00:04,044: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RWSD darumeru/ruOpenBookQA nlpcoreteam/ruMMLU 0.339 0.166 0.210 0.451 0.461 0.405 INFO: 2024-11-15 17:00:12,617: llmtf.base.evaluator: Starting eval on ['darumeru/rcb'] INFO: 2024-11-15 17:00:12,617: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 17:00:12,617: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 17:00:15,035: llmtf.base.darumeru/RCB: Loading Dataset: 2.42s INFO: 2024-11-15 17:00:17,696: llmtf.base.darumeru/RCB: Processing Dataset: 2.66s INFO: 2024-11-15 17:00:17,696: llmtf.base.darumeru/RCB: Results for darumeru/RCB: INFO: 2024-11-15 17:00:17,699: llmtf.base.darumeru/RCB: {'acc': 0.4681818181818182, 'f1_macro': 0.3981025874347421} INFO: 2024-11-15 17:00:17,700: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 17:00:17,701: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA nlpcoreteam/ruMMLU 0.354 0.166 0.210 0.433 0.451 0.461 0.405 INFO: 2024-11-15 17:00:26,448: llmtf.base.evaluator: Starting eval on ['darumeru/ruworldtree'] INFO: 2024-11-15 17:00:26,448: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 17:00:26,448: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 17:00:28,717: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 2.27s INFO: 2024-11-15 17:00:30,112: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 1.39s INFO: 2024-11-15 17:00:30,112: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: INFO: 2024-11-15 17:00:30,116: llmtf.base.darumeru/ruWorldTree: {'acc': 0.5619047619047619, 'f1_macro': 0.5621556341232892} INFO: 2024-11-15 17:00:30,116: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 17:00:30,116: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU 0.384 0.166 0.210 0.433 0.451 0.461 0.562 0.405 INFO: 2024-11-15 17:00:38,826: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] INFO: 2024-11-15 17:00:38,826: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 17:00:38,826: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 17:00:51,532: llmtf.base.daru/treewayextractive: Loading Dataset: 12.71s INFO: 2024-11-15 17:02:39,091: llmtf.base.daru/treewayextractive: Processing Dataset: 107.56s INFO: 2024-11-15 17:02:39,091: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: INFO: 2024-11-15 17:02:39,318: llmtf.base.daru/treewayextractive: {'r-prec': 0.3703670274170274} INFO: 2024-11-15 17:02:39,357: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 17:02:39,359: llmtf.base.evaluator: mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU 0.382 0.370 0.166 0.210 0.433 0.451 0.461 0.562 0.405 INFO: 2024-11-15 17:02:47,961: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] INFO: 2024-11-15 17:02:47,961: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 17:02:47,961: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 17:03:40,869: llmtf.base.daru/treewayabstractive: Processing Dataset: 230.11s INFO: 2024-11-15 17:03:40,869: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: INFO: 2024-11-15 17:03:40,870: llmtf.base.daru/treewayabstractive: {'rouge1': 0.30478241492489133, 'rouge2': 0.09260347651798656} INFO: 2024-11-15 17:03:40,872: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 17:03:40,872: llmtf.base.evaluator: mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU 0.362 0.199 0.370 0.166 0.210 0.433 0.451 0.461 0.562 0.405 INFO: 2024-11-15 17:04:33,322: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 105.36s INFO: 2024-11-15 17:07:11,362: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 158.04s INFO: 2024-11-15 17:07:11,362: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: INFO: 2024-11-15 17:07:11,425: llmtf.base.nlpcoreteam/enMMLU: metric subject abstract_algebra 0.320000 anatomy 0.481481 astronomy 0.480263 business_ethics 0.470000 clinical_knowledge 0.573585 college_biology 0.506944 college_chemistry 0.310000 college_computer_science 0.380000 college_mathematics 0.290000 college_medicine 0.485549 college_physics 0.303922 computer_security 0.640000 conceptual_physics 0.489362 econometrics 0.412281 electrical_engineering 0.551724 elementary_mathematics 0.402116 formal_logic 0.285714 global_facts 0.350000 high_school_biology 0.541935 high_school_chemistry 0.453202 high_school_computer_science 0.670000 high_school_european_history 0.587879 high_school_geography 0.621212 high_school_government_and_politics 0.616580 high_school_macroeconomics 0.443590 high_school_mathematics 0.348148 high_school_microeconomics 0.529412 high_school_physics 0.344371 high_school_psychology 0.642202 high_school_statistics 0.430556 high_school_us_history 0.573529 high_school_world_history 0.594937 human_aging 0.542601 human_sexuality 0.534351 international_law 0.619835 jurisprudence 0.555556 logical_fallacies 0.558282 machine_learning 0.366071 management 0.582524 marketing 0.782051 medical_genetics 0.430000 miscellaneous 0.579821 moral_disputes 0.546243 moral_scenarios 0.241341 nutrition 0.562092 philosophy 0.517685 prehistory 0.518519 professional_accounting 0.414894 professional_law 0.365711 professional_medicine 0.367647 professional_psychology 0.446078 public_relations 0.536364 security_studies 0.538776 sociology 0.592040 us_foreign_policy 0.700000 virology 0.433735 world_religions 0.631579 INFO: 2024-11-15 17:07:11,433: llmtf.base.nlpcoreteam/enMMLU: metric subject STEM 0.434923 humanities 0.507447 other (business, health, misc.) 0.503999 social sciences 0.551074 INFO: 2024-11-15 17:07:11,438: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.4993605425093448} INFO: 2024-11-15 17:07:11,470: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 17:07:11,472: llmtf.base.evaluator: mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU 0.376 0.199 0.370 0.166 0.210 0.433 0.451 0.461 0.562 0.499 0.405 INFO: 2024-11-15 17:07:20,297: llmtf.base.evaluator: Starting eval on ['darumeru/cp_para_ru'] INFO: 2024-11-15 17:07:20,298: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] INFO: 2024-11-15 17:07:20,298: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-11-15 17:07:22,739: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 2.44s INFO: 2024-11-15 17:10:45,613: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 202.87s INFO: 2024-11-15 17:10:45,613: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: INFO: 2024-11-15 17:10:45,614: llmtf.base.darumeru/cp_para_ru: {'tokens_per_word': 1.8729616000663136, 'symbol_per_token': 4.043905520913176, 'len': 0.927490542336371, 'lcs': 0.05} INFO: 2024-11-15 17:10:45,615: llmtf.base.evaluator: Ended eval INFO: 2024-11-15 17:10:45,615: llmtf.base.evaluator: mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU 0.346 0.199 0.370 0.166 0.210 0.433 0.451 0.050 0.461 0.562 0.499 0.405