The official repository of "Not All Metrics Are Guilty: Improving NLG benchmarks by Diversifying References".
The references and diversified references can be found in dataset/XXX/xxxx.reference and dataset/XXX/xxxx.reference.diversified.
git clone https://github.com/google-research/mt-metrics-eval.git
cd mt-metrics-eval
pip install .
cd ..
git clone https://github.com/google-research/bleurt.git
cd bleurt
pip install .
cd ..
git clone https://github.com/thompsonb/prism
cd prism
pip install -r requirements.txt
cd ..
pip install -U git+https://github.com/pltrdy/pyrouge
git clone https://github.com/pltrdy/files2rouge.git
cd files2rouge
python setup_rouge.py
python setup.py install
cd ..
git clone https://github.com/neulab/BARTScore
git clone https://github.com/AIPHES/emnlp19-moverscore
pip install sacrebleu, bert-score, unbabel-comet, pycocoevalcapDownload data:
alias mtme='python3 -m mt_metrics_eval.mtme'
mtme --downloadPrepare data:
python data.py zh-enCalculate metric scores for different metrics:
python score.py score/zh-en.json chrf/bleu/bertscore/bleurt/prism/comet/bartscore/gembaCalculate the correlation scores for different metrics:
python analysis.py score/zh-en.json chrf/bleu/bertscore/bleurt/prism/comet/bartscore/gembaDownload data:
wget https://storage.googleapis.com/sfr-summarization-repo-research/model_annotations.aligned.jsonlPrepare data:
python data.pyCalculate metric scores for different metrics:
python score.py score/summeval.json rouge/bertscore/moverscore/chatgptCalculate the correlation scores for different metrics:
python analysis.py score/summeval.json rouge/bertscore/moverscore/chatgptDownload data:
wget https://filebox.ece.vt.edu/~vrama91/CIDEr_miscellanous/cider_datasets.tarPrepare data:
python data.pyCalculate metric scores for different metrics:
python score.py score/pascal.json bleu/rouge/meteor/cider/spice/bertscoreCalculate the correlation scores for different metrics:
python analysis.py score/pascal.json bleu/rouge/meteor/cider/spice/bertscore