Spaces:

evaluate-metric

/

sari

like 7

Running

Hi everyone, I noticed that this metric outputs different scores than the EASSE corpus_sari function. For example, this:

from easse.sari import corpus_sari
c = corpus_sari(orig_sents=["About 95 species are currently accepted.", "The cat perched on the mat."],  
            sys_sents=["About 95 you now get in.", "Cat on mat."], 
            refs_sents=[["About 95 species are currently known.", "The cat sat on the mat."],
                        ["About 95 species are now accepted.", "The cat is on the mat."],  
                        ["95 species are now accepted.", "The cat sat."]])

gives a sari of 33.17472563619544, whereas this:

import evaluate
sari = evaluate.load("sari")
sources=["About 95 species are currently accepted.", "The cat perched on the mat."]
predictions=["About 95 you now get in.", "Cat on mat."]
references=[["About 95 species are currently known.", "About 95 species are now accepted.", "95 species are now accepted.", ],
                        ["The cat sat on the mat.", "The cat is on the mat.", "The cat sat."]]
sari_score = sari.compute(sources=sources, predictions=predictions, references=references)

returns a sari of 29.13266517433184.

However, if the source and the output sentences are identical, the evaluate sari is much higher than the easse sari. Here:

from easse.sari import corpus_sari
c = corpus_sari(orig_sents=["About 95 species are currently accepted."],  
            sys_sents=["About 95 species are currently accepted."], 
            refs_sents=[["About 95 species are currently known."],
                        ["About 95 species are now accepted."],  
                        ["95 species are now accepted."]])

I got a sari of 21.873217526575058, and here:

import evaluate
sari = evaluate.load("sari")
sources=["About 95 species are currently accepted."]
predictions=["About 95 species are currently accepted."]
references=[["About 95 species are currently known.", "About 95 species are now accepted.", "95 species are now accepted.", ]]

I got a sari of 55.20655085990839.

Am I doing something wrong? Why are the scores so different?