SUGARCREPE++

The SUGARCREPE++ dataset is created to evaluate the sensitivity of vision language models (VLMs) and unimodal language models (ULMs) to semantic and lexical alterations. The SUGARCREPE dataset consists of (only) one positive and one hard negative caption for each image. Relative to the negative caption, a single positive caption can either have low or high lexical overlap. The original SUGARCREPE only captures the high overlap case. To evaluate the sensitivity of encoded semantics to lexical alteration, we require an additional positive caption with a different lexical composition. SUGARCREPE++ fills this gap by adding an additional positive caption enabling a more thorough assessment of models’ abilities to handle semantic content and lexical variation.

📄 Paper preprint: SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations

We also host the SUGARCREPE++ dataset in Hugging Face dataset here.

Vision-Language model evaluation on SUGARCREPE++

We evaluate a comprehensive list of Vision-Language Models (VLMs) on SUGARCREPE++. We evaluate VLMs under two different settings: (1) image-text task (ITT) and (2) text-only task (TOT). As explained in our paper, in ITT, both the image and the corresponding triplet of captions (two positive captions and one negative caption) are provided as input. In TOT, only the text encoder of the VLMs is evaluated using the triplet of captions. Please refer to the VLMs/README.md in the VLMs folder for the steps to reproduce the results in the paper.

Unimodal Language model evaluation in SUGARCREPE++

We designed the SUGARCREPE++ dataset such that the overlap of semantic information between the two positive captions is always higher than between the positive and negative captions, even without considering the image. We use the text-only task (TOT) metric for ULMs as defined in the paper.

ULMs can be evaluated with the following steps:

1. Setting up the Python environment for ULMs:

   pip install -r ulm-requirements.txt

2. This will download and evaluate models that can be run on a medium-sized GPU.

   python evaluate_ulms.py --data_dir data --models modelset1

3. This will download and evaluate models that can be inferred on a 40GB GPU.

   python evaluate_ulms.py --data_dir data --models modelset2 modelset3

Note: For the Llama model, you need to request access from here here; more info can be found here.

4. For help, use python evaluate_ulms.py --help

5. The ULM evaluation run will generate the following in the current directory.

   • Files:
     • ulm_results.log contains the results of the evaluation.
     • ulm_model_stats.log contains model size and embedding size information of the evaluated ULMs.
   • Directory:
     • ulm-results contains pickle files with the sample-wise and aggregate results for each model/dataset.
     • ulm_model_data contains the downloaded weights from huggingface. This may be modified by changing the CACHE_DIR in evaluate_ulms.py

6. To summarize results of a previous run that is stored in DIR, run the following command.

   python -c 'from evaluate_ulms import summarize_results; print(summarize_results("DIR"))'

SUGARCREPE++ Generation Pipeline

We generate an extra caption using the Mistral model and human-validate the generated captions.

The steps to generate extra positive captions are shown below:

1. Download the original SUGARCREPE dataset from here.

2. Download the mistral-7b model here.

3. Run the following script.

   python generate_sugarcrepe_plus-mistral.py --data_dir {directory to original sugarcrepe}

4. The scripts will run both stages of the generation pipeline and create a new directory called data/sugarcrepe-plus-plus-mistral with outputs from both stages. The 'checked_caption' are automatically validated captions which are further considered for human validation.

Note: Further human validation would be required to match the quality of SUGARCREPE++. The above output files would be similar in quality to files in generated_data.

---

Contact

For further assistance, email:

aman.jaiswal@dal.ca

sriharsha.d@dal.ca

📎 Citation

@misc{dumpala2024sugarcrepe,
    title={SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations},
    author={Sri Harsha Dumpala and Aman Jaiswal and Chandramouli Sastry and Evangelos Milios and Sageev Oore and Hassan Sajjad},
    year={2024},
    eprint={2406.11171},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

License

Shield:

This work is licensed under a Creative Commons Attribution 4.0 International License.