WorldCuisines: a massive-scale benchmark for multilingual and multicultural visual question answering on global cuisines
A massive multilingual and multicultural visual question answering benchmark for evaluating VLMs on global cuisines.
Vision Language Models (VLMs) often struggle with culture-specific knowledge, particularly in languages other than English and in underrepresented cultural contexts. To evaluate their understanding of such knowledge, we introduce WorldCuisines, a massive-scale benchmark for multilingual and multicultural, visually grounded language understanding. This benchmark includes a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects, spanning 9 language families and featuring over 1 million data points, making it the largest multicultural VQA benchmark to date. It includes tasks for identifying dish names and their origins. We provide evaluation datasets in two sizes (12k and 60k instances) alongside a training dataset (1 million instances). Our findings show that while VLMs perform better with correct location context, they struggle with adversarial contexts and predicting specific regional cuisines and languages. To support future research, we release a knowledge base with annotated food entries and images along with the VQA data.
Latest publications
KODIS: A multicultural dispute resolution dialogue corpus
A dyadic dispute resolution corpus containing thousands of dialogues from over 75 countries.
NAACLProxyLM: predicting language model performance on multilingual tasks via proxy models
A scalable framework using proxy models to efficiently predict the performance of multilingual language models on NLP tasks.
NAACLBuilding safe GenAI applications: An end-to-end overview of red teaming for Large Language Models
A survey covering attack methods, evaluation, metrics, and tools for identifying and mitigating GenAI application vulnerabilities.
NAACL