⚖️ Gender Biases in Automatic Evaluation Metrics for Image Captioning

Haoyi Qiu¹, Zi-Yi Dou¹, Tianlu Wang², Asli Celikyilmaz², Nanyun Peng^1
1UCLA, ²Meta FAIR

🫧 Overview

Model-based evaluation metrics (e.g., CLIPScore and GPTScore) have demonstrated decent correlations with human judgments in various language generation tasks. However, their impact on fairness remains largely unexplored. It is widely recognized that pretrained models can inadvertently encode societal biases, thus employing these models for evaluation purposes may inadvertently perpetuate and amplify biases.

For example, an evaluation metric may favor the caption “a woman is calculating an account book” over “a man is calculating an account book,” even if the image only shows male accountants.

In this work:

• We conduct a systematic study of gender biases in model-based automatic evaluation metrics for image captioning tasks.
• We curate a dataset comprising profession, activity, and object concepts associated with stereotypical gender associations.
• We demonstrate the negative consequences of using these biased metrics, including the inability to differentiate between biased and unbiased generations, as well as the propagation of biases to generation models through reinforcement learning.
• We present a simple and effective way to mitigate the metric bias without hurting the correlations with human judgments. Our dataset and framework lay the foundation for understanding the potential harm of model-based evaluation metrics, and facilitate future works to develop more inclusive evaluation metrics.

📚 PAO-EvalBias Dataset

We collect images of people with various professions, activities, and objects (PAO-EvalBias). For each concept in the lexicons, we use templates to construct one reference as well as two candidates containing the correct and incorrect gender, denoted as the good and bad captions respectively. Our approach involves pairing a gender from protected groups (man or woman) with a concept in professions, activities, or objects. We collect 91,640 images for PAO-EvalBias including 🧑‍💼 88 professions, 🏃‍♀️ 52 activities, and 🍎 39 objects.

Figure 1: Example instances from PAO-EvalBias.

Candidate and reference captions follow specific patterns described in the table below. The lexicon word is highlighted in bold in the reference caption, while the gender identification word is in blue for a good caption and in red for a bad caption. A good caption maintains the same gender as the reference sentence, while a bad caption replaces the gender in the good caption with an incorrect gender. For example, in the image located at the top left corner featuring a male editor, the good caption reads “a man who is an editor,” while the bad caption replaces “man” with “woman”.

Figure 2: Caption patterns in PAO-EvalBias.

📦 Dataset Download

You can download the images dataset from https://drive.google.com/file/d/1Aiby0oakEMEOaoGnm8zZQIwwKDghMUj1/view?usp=drive_link.

You can find all captions for the images under the directory captions. Each entry is formatted as:

  {
    "image_path": "../PAO-EvalBias-images/woman_activity/baking/woman_baking_4_1.jpg",
    "reference": "a photo of a woman who is baking",
    "good_caption": "a woman who is baking",
    "bad_caption": "a man who is baking"
  }

✨ Citation

If you find this work useful, please consider citing our paper as:

@inproceedings{qiu2023gender,
  title={Gender Biases in Automatic Evaluation Metrics for Image Captioning},
  author={Qiu, Haoyi and Dou, Zi-Yi and Wang, Tianlu and Celikyilmaz, Asli and Peng, Nanyun},
  booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  rank={20},
  keyword={evaluation, fairness},
  paper_url={https://arxiv.org/abs/2305.14268},
  year={2023}
}