Are Pretrained Multilingual Models Equally Fair across Languages?
Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt
Standard
Are Pretrained Multilingual Models Equally Fair across Languages? / Cabello Piqueras, Laura; Søgaard, Anders.
Proceedings of the 29th International Conference on Computational Linguistics. International Committee on Computational Linguistics, 2022. s. 3597–3605.Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - Are Pretrained Multilingual Models Equally Fair across Languages?
AU - Cabello Piqueras, Laura
AU - Søgaard, Anders
N1 - Conference code: 29
PY - 2022
Y1 - 2022
N2 - Pretrained multilingual language models can help bridge the digital language divide, enabling high-quality NLP models for lower-resourced languages. Studies of multilingual models have so far focused on performance, consistency, and cross-lingual generalisation. However, with their wide-spread application in the wild and downstream societal impact, it is important to put multilingual models under the same scrutiny as monolingual models. This work investigates the group fairness of multilingual models, asking whether these models are equally fair across languages. To this end, we create a new four-way multilingual dataset of parallel cloze test examples (MozArt), equipped with demographic information (balanced with regard to gender and native tongue) about the test participants. We evaluate three multilingual models on MozArt –mBERT, XLM-R, and mT5– and show that across the four target languages, the three models exhibit different levels of group disparity, e.g., exhibiting near-equal risk for Spanish, but high levels of disparity for German.
AB - Pretrained multilingual language models can help bridge the digital language divide, enabling high-quality NLP models for lower-resourced languages. Studies of multilingual models have so far focused on performance, consistency, and cross-lingual generalisation. However, with their wide-spread application in the wild and downstream societal impact, it is important to put multilingual models under the same scrutiny as monolingual models. This work investigates the group fairness of multilingual models, asking whether these models are equally fair across languages. To this end, we create a new four-way multilingual dataset of parallel cloze test examples (MozArt), equipped with demographic information (balanced with regard to gender and native tongue) about the test participants. We evaluate three multilingual models on MozArt –mBERT, XLM-R, and mT5– and show that across the four target languages, the three models exhibit different levels of group disparity, e.g., exhibiting near-equal risk for Spanish, but high levels of disparity for German.
M3 - Article in proceedings
SP - 3597
EP - 3605
BT - Proceedings of the 29th International Conference on Computational Linguistics
PB - International Committee on Computational Linguistics
T2 - THE 29TH INTERNATIONAL CONFERENCE ON COMPUTATIONAL LINGUISTICS
Y2 - 12 October 2022 through 17 October 2022
ER -
ID: 341498752