Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.29968

CulturalRiddles: A Multicultural Benchmark of Riddles

My Chiffon Nguyen·Tianyi Hu·Vinod Anbalagan·Henry Gagnier·Pouya Sadeghi·Farah Abdou·Tidor-Vlad Pricope·Diego MMS Oliveira·Jocelyn K D'Arcy·Madeline Smith·Oluwaseun A Ajayi·Yusser Al Ghussin·Zena Al-Khalili·Micol Altomare·Shayan Amir·Ahanaf Aziz·Bryan Chen·Ruchira Dhar·Abdellah El Mekki·Rawan Mohammed Elframawy·Akhil Eppa·Silvia Fernandez·Alex Gimei·Arya Hariharan·Christian Hoang·RamMohanRao Kadiyala·Diana Kozachek·Peerat Limkonchotiwat·Ankita Maity·Di Mo·Ayush Munot·Aylin Naebzadeh·Anastasiia Nikitina·Dina Pisarevskaya·Divas Poudel·Jebish Purbey·Md Rabius Sany Apu·Ahmad Mufli Ramadhan·Ralitza Nikolova Rizova·Diogo M. Serrano-Miranda·Vivek Gopalkrishna Silimkhan·Luis Fernando Solis Navarro·Kaiser Sun·Arya Suneesh·Manaswitha Sunkara·Tejaswi Sunkara·Jonathan E Swindell·Hanna Yukhymenko·Jingyi Zheng·Esther Adenuga·Ilia Afanasev·Wafa Aissa·Lisa Alazraki·Lauren C Altomare·Priscilla Amondi Amuok·Emile Anand·Gabriel Assis·José Matías Avalos·Silvia Lorena Avila·Zakaria Baannou·Niyati Bafna·Yash Bagla·Sarah Bahrami·Bontu F Balcha·Vanya Bannihatti Kumar·Irene Baucells·Abdumutalib Begmuratov·Naman Bhatia·Souvik Bhattacharyya·Isabel Blanco Ferreiro·Felipe Bochini·Sayuru Bopitiya·Angana Borah·Tulika Bose·Olena Bugaiova·Juliana Carvalho·Nancy F Chen·Stephy Chen·Rochelle Choenni·Jayanth Krishna Chundru·Leandro A Cossio·Nebojša Damnjanović·Ona de Gibert·Marco De Santis·Pedro Delgado·Kaustubh Dholé·Adnan El Assadi·Wesam El-Sayed·Nadine M Elframawy·Oumayma Essarhi·Naome A Etori·Reuben Chagas Fernandes·Saanidhi Pravin Gade·Javier García Gilabert·Álvaro González-Escapa·Bin Hu·Kaiyu Huang·Abderrahmane Issam·Danka Jokić·Ulugbek Khalilov
View original source ↗Read source document ↗Export metadata ↓

Abstract

As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.

Explore related subjects

cs.CLcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego MMS Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy, Akhil Eppa, Silvia Fernandez, Alex Gimei, Arya Hariharan, Christian Hoang, RamMohanRao Kadiyala, Diana Kozachek, Peerat Limkonchotiwat, Ankita Maity, Di Mo, Ayush Munot, Aylin Naebzadeh, Anastasiia Nikitina, Dina Pisarevskaya, Divas Poudel, Jebish Purbey, Md Rabius Sany Apu, Ahmad Mufli Ramadhan, Ralitza Nikolova Rizova, Diogo M. Serrano-Miranda, Vivek Gopalkrishna Silimkhan, Luis Fernando Solis Navarro, Kaiser Sun, Arya Suneesh, Manaswitha Sunkara, Tejaswi Sunkara, Jonathan E Swindell, Hanna Yukhymenko, Jingyi Zheng, Esther Adenuga, Ilia Afanasev, Wafa Aissa, Lisa Alazraki, Lauren C Altomare, Priscilla Amondi Amuok, Emile Anand, Gabriel Assis, José Matías Avalos, Silvia Lorena Avila, Zakaria Baannou, Niyati Bafna, Yash Bagla, Sarah Bahrami, Bontu F Balcha, Vanya Bannihatti Kumar, Irene Baucells, Abdumutalib Begmuratov, Naman Bhatia, Souvik Bhattacharyya, Isabel Blanco Ferreiro, Felipe Bochini, Sayuru Bopitiya, Angana Borah, Tulika Bose, Olena Bugaiova, Juliana Carvalho, Nancy F Chen, Stephy Chen, Rochelle Choenni, Jayanth Krishna Chundru, Leandro A Cossio, Nebojša Damnjanović, Ona de Gibert, Marco De Santis, Pedro Delgado, Kaustubh Dholé, Adnan El Assadi, Wesam El-Sayed, Nadine M Elframawy, Oumayma Essarhi, Naome A Etori, Reuben Chagas Fernandes, Saanidhi Pravin Gade, Javier García Gilabert, Álvaro González-Escapa, Bin Hu, Kaiyu Huang, Abderrahmane Issam, Danka Jokić, Ulugbek Khalilov. 2026-10-21. CulturalRiddles: A Multicultural Benchmark of Riddles. https://arxiv.org/abs/2610.29968

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence

Language models offer a way to scale costly, labour-intensive expert annotation, but their ability to reproduce expert judgements remains uncertain. This is particularly challenging in law, where annotators must distinguish the reasoning supporting judicial decisions from other legally relevant discussion. We present AnnoCaseLaw as a densely annotated benchmark comprising $471$ U.S. civil-negligence court opinions with more than $12,000$ expert highlights across five annotation categories. We evaluate proprietary frontier, open-weight, legal-specialised, and decision models on their ability to reproduce these annotations. Models perform better on Facts and Outcome than on Application of Law to Facts, Procedural History, and Relevant Precedents. On Application of Law to Facts, proprietary frontier models achieve the strongest balance of reasoning coverage and precision, while the open-weight model Gemma 4 achieves comparable Macro-F1 to proprietary frontier models at a fraction of the estimated inference cost. Our findings highlight the importance of evaluating annotation coverage and cost alongside aggregate scores when assessing the feasibility of automated legal annotation.

cs.CL↗
arXiv2026

Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors

This position paper argues that many behaviours framed as either "stochastic parroting" or "advanced reasoning" are better explained as context-directed extrapolation from training-data priors, and that ML and NLP research communities would benefit from evaluations that distinguish remembering and basic extrapolation from genuine application-level generalisation. Our central contribution is an actionable evaluation checklist and design patterns that make these distinctions testable in practice. Our framework posits that LLMs extrapolate from training data priors while using context to guide the model to the appropriate priors; we call this "context-directed extrapolation." Specifically, this context direction is achieved through examples in base models, leading to in-context learning, while instruction tuning allows analogous task specification via prompts rather than explicit examples. We argue that adopting this more realistic framework and controls for improving evaluation would better calibrate claims about LLM "reasoning," refocus research on how LLM performance interacts with training data and generalisability to out-of-domain tasks, and reduce miscalibration in deployment and policy decisions.

cs.CL↗
arXiv2026

Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games

Large Language Models (LLMs) have shown potential in simulating human behaviors and performing theory-of-mind (ToM) reasoning, crucial for complex social interactions. We investigate ToM reasoning's role in aligning agentic behaviors with human norms in negotiation tasks, using the ultimatum game as our referenced task. We initialized LLM agents with different prosocial beliefs (Greedy, Fair, Selfless) and reasoning methods (chain of thought and ToM reasoning of varying levels), examining their decision-making processes and outcomes across multiple LLMs, including reasoning models like o3-mini and DeepSeek-R1 Distilled Qwen 32B. We perform 2,700 simulations to show that ToM reasoning enhances behavioral alignment with human norms, decision-making consistency, and negotiation outcomes. Consistent with prior findings, reasoning LLMs exhibit limited capability compared to ToM-enhanced LLMs, with different game roles benefiting from different ToM orders. Agents almost always chose strategies consistent with their assigned beliefs, and doing so aligned better with human norms in nearly every role, yet Greedy and Selfless agents still fell short of their norms in outcomes. Human verification further revealed that Llama 3.3 70B produces reasoning most consistent with its actions and beliefs. Our findings advance understanding of ToM's role in human-AI interaction and cooperative decision-making. The code used for our experiments can be found at https://github.com/Stealth-py/UltimatumToM.

cs.CL↗