arXiv subjects

Walid Magdy

Publications and source records attributed to Walid Magdy.

At least 19 recordsLinked to original sources

arXiv2026

IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge

Large language models are increasingly consulted for Islamic knowledge, yet no comprehensive benchmark evaluates their performance across core Islamic disciplines. We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (jurisprudence, 4,000 questions). Each track is formed of multiple types of questions to examine LLMs capabilities handling different aspects of Islamic knowledge. The benchmark is used to create the IslamicMMLU public leaderboard for evaluating LLMs, and we initially evaluate 26 LLMs, where their averaged accuracy across the three tracks varied between 39.8% to 93.8% (by Gemini 3 Flash). The Quran track shows the widest span (99.3% to 32.4%), while the Fiqh track includes a novel madhab (Islamic school of jurisprudence) bias detection task revealing variable school-of-thought preferences across models. Arabic-specific models show mixed results, but they all underperform compared to frontier models. The evaluation code and leaderboard are made publicly available.

cs.CL↗

arXiv2025

Fabricating Holiness: Characterizing Religious Misinformation Circulators on Arabic Social Media

Misinformation is a growing concern in a decade involving critical global events. While social media regulation is mainly dedicated towards the detection and prevention of fake news and political misinformation, there is limited research about religious misinformation which has only been addressed through qualitative approaches. In this work, we study the spread of fabricated quotes (Hadith) that are claimed to belong to Prophet Muhammad (the prophet of Islam) as a case study demonstrating one of the most common religious misinformation forms on Arabic social media. We attempt through quantitative methods to understand the characteristics of social media users who interact with fabricated Hadith. We spotted users who frequently circulate fabricated Hadith and others who frequently debunk it to understand the main differences between the two groups. We used Logistic Regression to automatically predict their behaviors and analyzed its weights to gain insights about the characteristics and interests of each group. We find that both fabricated Hadith circulators and debunkers have generally a lot of ties to religious accounts. However, circulators are identified by many accounts that follow the Shia branch of Islam, Sunni Islamic public figures from the gulf countries, and many Sunni non-professional pages posting Islamic content. On the other hand, debunkers are identified by following academic Islamic scholars from multiple countries and by having more intellectual non-religious interests like charity, politics, and activism.

cs.SI↗

arXiv2025

SocioXplorer: An Interactive Tool for Topic and Network Analysis in Social Data

SocioXplorer is a powerful interactive tool that computational social science researchers can use to understand topics and networks in social data from Twitter (X) and YouTube. It integrates, among other things, artificial intelligence, natural language processing and social network analysis. It can be used with ``live" datasets that receive regular updates. SocioXplorer is an extension of a previous system called TwiXplorer, which was limited to the analysis of archival Twitter (X) data. SocioXplorer builds on this by adding the ability to analyse YouTube data, greater depth of analysis and batch data processing. We release it under the Apache 2 licence.

cs.SI↗

arXiv2025

Culture Matters in Toxic Language Detection in Persian

Toxic language detection is crucial for creating safer online environments and limiting the spread of harmful content. While toxic language detection has been under-explored in Persian, the current work compares different methods for this task, including fine-tuning, data enrichment, zero-shot and few-shot learning, and cross-lingual transfer learning. What is especially compelling is the impact of cultural context on transfer learning for this task: We show that the language of a country with cultural similarities to Persian yields better results in transfer learning. Conversely, the improvement is lower when the language comes from a culturally distinct country. Warning: This paper contains examples of toxic language that may disturb some readers. These examples are included for the purpose of research on toxic detection.

cs.CL↗

arXiv2025

Revisiting Common Assumptions about Arabic Dialects in NLP

Arabic has diverse dialects, where one dialect can be substantially different from the others. In the NLP literature, some assumptions about these dialects are widely adopted (e.g., ``Arabic dialects can be grouped into distinguishable regional dialects") and are manifested in different computational tasks such as Arabic Dialect Identification (ADI). However, these assumptions are not quantitatively verified. We identify four of these assumptions and examine them by extending and analyzing a multi-label dataset, where the validity of each sentence in 11 different country-level dialects is manually assessed by speakers of these dialects. Our analysis indicates that the four assumptions oversimplify reality, and some of them are not always accurate. This in turn might be hindering further progress in different Arabic NLP tasks.

cs.CL↗

arXiv2025

Who Should Set the Standards? Analysing Censored Arabic Content on Facebook during the Palestine-Israel Conflict

Nascent research on human-computer interaction concerns itself with fairness of content moderation systems. Designing globally applicable content moderation systems requires considering historical, cultural, and socio-technical factors. Inspired by this line of work, we investigate Arab users' perception of Facebook's moderation practices. We collect a set of 448 deleted Arabic posts, and we ask Arab annotators to evaluate these posts based on (a) Facebook Community Standards (FBCS) and (b) their personal opinion. Each post was judged by 10 annotators to account for subjectivity. Our analysis shows a clear gap between the Arabs' understanding of the FBCS and how Facebook implements these standards. The study highlights a need for discussion on the moderation guidelines on social media platforms about who decides the moderation guidelines, how these guidelines are interpreted, and how well they represent the views of marginalised user communities.

cs.SI↗

arXiv2025

Fanar: An Arabic-Centric Multimodal Generative AI Platform

We present Fanar, a platform for Arabic-centric multimodal generative AI systems, that supports language, speech and image generation tasks. At the heart of Fanar are Fanar Star and Fanar Prime, two highly capable Arabic Large Language Models (LLMs) that are best in the class on well established benchmarks for similar sized models. Fanar Star is a 7B (billion) parameter model that was trained from scratch on nearly 1 trillion clean and deduplicated Arabic, English and Code tokens. Fanar Prime is a 9B parameter model continually trained on the Gemma-2 9B base model on the same 1 trillion token set. Both models are concurrently deployed and designed to address different types of prompts transparently routed through a custom-built orchestrator. The Fanar platform provides many other capabilities including a customized Islamic Retrieval Augmented Generation (RAG) system for handling religious prompts, a Recency RAG for summarizing information about current or recent events that have occurred after the pre-training data cut-off date. The platform provides additional cognitive capabilities including in-house bilingual speech recognition that supports multiple Arabic dialects, voice and image generation that is fine-tuned to better reflect regional characteristics. Finally, Fanar provides an attribution service that can be used to verify the authenticity of fact based generated content. The design, development, and implementation of Fanar was entirely undertaken at Hamad Bin Khalifa University's Qatar Computing Research Institute (QCRI) and was sponsored by Qatar's Ministry of Communications and Information Technology to enable sovereign AI technology development.

cs.CL↗

arXiv2024

"The Prophet said so!": On Exploring Hadith Presence on Arabic Social Media

Hadith, the recorded words and actions of the prophet Muhammad, is a key source of the instructions and foundations of Islam, alongside the Quran. Interpreting individual hadiths and verifying their authenticity can be difficult, even controversial, and the subject has attracted the attention of many scholars who have established an entire science of Hadith criticism. Recent quantitative studies of hadiths focus on developing systems for automatic classification, authentication, and information retrieval that operate over existing hadith compilations. Qualitative studies on the other hand try to discuss different social and political issues from the perspective of hadiths, or they inspect how hadiths are used in specific contexts in official communications and press releases for argumentation and propaganda. However, there are no studies that attempt to understand the actual presence of hadiths among Muslims in their daily lives and interactions. In this study, we try to fill this gap by exploring the presence of hadiths on Twitter from January 2019 to January 2023. We highlight the challenges that quantitative methods should consider while processing texts that include hadiths and we provide a methodology for Islamic scholars to validate their hypotheses about hadiths on big data that better represent the position of the society and Hadith influence on it.

cs.SI↗

arXiv2024

NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task

We describe the findings of the fifth Nuanced Arabic Dialect Identification Shared Task (NADI 2024). NADI's objective is to help advance SoTA Arabic NLP by providing guidance, datasets, modeling opportunities, and standardized evaluation conditions that allow researchers to collaboratively compete on pre-specified tasks. NADI 2024 targeted both dialect identification cast as a multi-label task (Subtask~1), identification of the Arabic level of dialectness (Subtask~2), and dialect-to-MSA machine translation (Subtask~3). A total of 51 unique teams registered for the shared task, of whom 12 teams have participated (with 76 valid submissions during the test phase). Among these, three teams participated in Subtask~1, three in Subtask~2, and eight in Subtask~3. The winning teams achieved 50.57 F\textsubscript{1} on Subtask~1, 0.1403 RMSE for Subtask~2, and 20.44 BLEU in Subtask~3, respectively. Results show that Arabic dialect processing tasks such as dialect identification and machine translation remain challenging. We describe the methods employed by the participating teams and briefly offer an outlook for NADI.

cs.CL↗

arXiv2024

Estimating the Level of Dialectness Predicts Interannotator Agreement in Multi-dialect Arabic Datasets

On annotating multi-dialect Arabic datasets, it is common to randomly assign the samples across a pool of native Arabic speakers. Recent analyses recommended routing dialectal samples to native speakers of their respective dialects to build higher-quality datasets. However, automatically identifying the dialect of samples is hard. Moreover, the pool of annotators who are native speakers of specific Arabic dialects might be scarce. Arabic Level of Dialectness (ALDi) was recently introduced as a quantitative variable that measures how sentences diverge from Standard Arabic. On randomly assigning samples to annotators, we hypothesize that samples of higher ALDi scores are harder to label especially if they are written in dialects that the annotators do not speak. We test this by analyzing the relation between ALDi scores and the annotators' agreement, on 15 public datasets having raw individual sample annotations for various sentence-classification tasks. We find strong evidence supporting our hypothesis for 11 of them. Consequently, we recommend prioritizing routing samples of high ALDi scores to native speakers of each sample's dialect, for which the dialect could be automatically identified at higher accuracies.

cs.CL↗

arXiv2023

Arabic Dialect Identification under Scrutiny: Limitations of Single-label Classification

Automatic Arabic Dialect Identification (ADI) of text has gained great popularity since it was introduced in the early 2010s. Multiple datasets were developed, and yearly shared tasks have been running since 2018. However, ADI systems are reported to fail in distinguishing between the micro-dialects of Arabic. We argue that the currently adopted framing of the ADI task as a single-label classification problem is one of the main reasons for that. We highlight the limitation of the incompleteness of the Dialect labels and demonstrate how it impacts the evaluation of ADI systems. A manual error analysis for the predictions of an ADI, performed by 7 native speakers of different Arabic dialects, revealed that $\approx$ 66% of the validated errors are not true errors. Consequently, we propose framing ADI as a multi-label classification task and give recommendations for designing new ADI datasets.

cs.CL↗

arXiv2023

ALDi: Quantifying the Arabic Level of Dialectness of Text

Transcribed speech and user-generated text in Arabic typically contain a mixture of Modern Standard Arabic (MSA), the standardized language taught in schools, and Dialectal Arabic (DA), used in daily communications. To handle this variation, previous work in Arabic NLP has focused on Dialect Identification (DI) on the sentence or the token level. However, DI treats the task as binary, whereas we argue that Arabic speakers perceive a spectrum of dialectness, which we operationalize at the sentence level as the Arabic Level of Dialectness (ALDi), a continuous linguistic variable. We introduce the AOC-ALDi dataset (derived from the AOC dataset), containing 127,835 sentences (17% from news articles and 83% from user comments on those articles) which are manually labeled with their level of dialectness. We provide a detailed analysis of AOC-ALDi and show that a model trained on it can effectively identify levels of dialectness on a range of other corpora (including dialects and genres not included in AOC-ALDi), providing a more nuanced picture than traditional DI systems. Through case studies, we illustrate how ALDi can reveal Arabic speakers' stylistic choices in different situations, a useful property for sociolinguistic analyses.

cs.CL↗

arXiv2023

DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Models

A few benchmarking datasets have been released to evaluate the factual knowledge of pretrained language models. These benchmarks (e.g., LAMA, and ParaRel) are mainly developed in English and later are translated to form new multilingual versions (e.g., mLAMA, and mParaRel). Results on these multilingual benchmarks suggest that using English prompts to recall the facts from multilingual models usually yields significantly better and more consistent performance than using non-English prompts. Our analysis shows that mLAMA is biased toward facts from Western countries, which might affect the fairness of probing models. We propose a new framework for curating factual triples from Wikidata that are culturally diverse. A new benchmark DLAMA-v1 is built of factual triples from three pairs of contrasting cultures having a total of 78,259 triples from 20 relation predicates. The three pairs comprise facts representing the (Arab and Western), (Asian and Western), and (South American and Western) countries respectively. Having a more balanced benchmark (DLAMA-v1) supports that mBERT performs better on Western facts than non-Western ones, while monolingual Arabic, English, and Korean models tend to perform better on their culturally proximate facts. Moreover, both monolingual and multilingual models tend to make a prediction that is culturally or geographically relevant to the correct label, even if the prediction is wrong.

cs.CL↗

arXiv2022

Twitter has a Binary Privacy Setting, are Users Aware of How It Works?

Twitter accounts are public by default, but Twitter gives the option to create protected accounts, where only approved followers can see their tweets. The publicly visible information changes based on the account type and the visibility of tweets also depends solely on the poster's account type which can cause unintended disclosures especially when users interact. We surveyed 336 Twitter users to understand users' awareness of account information visibility, as well as the tweet visibility when users interact. We find that our participants are aware of the visibility of their profile information and individual tweets. However, the visibility of followed topics, lists, and interactions with protected accounts is confusing. Only 31% of the participants were aware that a reply by a public account to a protected account's tweet would be publicly visible. Surprisingly, having a protected account does not result in a better understanding of the account information or tweet visibility.

cs.SI↗

arXiv2022

AX-MABSA: A Framework for Extremely Weakly Supervised Multi-label Aspect Based Sentiment Analysis

Aspect Based Sentiment Analysis is a dominant research area with potential applications in social media analytics, business, finance, and health. Prior works in this area are primarily based on supervised methods, with a few techniques using weak supervision limited to predicting a single aspect category per review sentence. In this paper, we present an extremely weakly supervised multi-label Aspect Category Sentiment Analysis framework which does not use any labelled data. We only rely on a single word per class as an initial indicative information. We further propose an automatic word selection technique to choose these seed categories and sentiment words. We explore unsupervised language model post-training to improve the overall performance, and propose a multi-label generator model to generate multiple aspect category-sentiment pairs per review sentence. Experiments conducted on four benchmark datasets showcase our method to outperform other weakly supervised baselines by a significant margin.

cs.CL↗

arXiv2022

Don't Take it Personally: Analyzing Gender and Age Differences in Ratings of Online Humor

Computational humor detection systems rarely model the subjectivity of humor responses, or consider alternative reactions to humor - namely offense. We analyzed a large dataset of humor and offense ratings by male and female annotators of different age groups. We find that women link these two concepts more strongly than men, and they tend to give lower humor ratings and higher offense scores. We also find that the correlation between humor and offense increases with age. Although there were no gender or age differences in humor detection, women and older annotators signalled that they did not understand joke texts more often than men. We discuss implications for computational humor detection and downstream tasks.

cs.CL↗

arXiv2022

Understanding Privacy Switching Behaviour on Twitter

Changing a Twitter account's privacy setting between public and protected changes the visibility of past tweets. By inspecting the privacy setting of over 100K Twitter users over 3 months, we noticed that over 40% of those users change their privacy setting at least once with around 16% changing it over 5 times. This motivated us to explore the reasons why people switch their privacy setting. We studied these switching phenomena quantitatively by comparing the tweeting behaviour of users when public vs protected, and qualitatively using two follow-up surveys (n=100, n=324) to understand potential reasoning behind the observed behaviours. Our quantitative analysis shows that users who switch privacy settings mention others and share hashtags more when their setting is public. Our surveys highlighted that users turn protected to share personal content and regulate boundaries while they turn public to interact with others in ways prevented by being protected.

cs.HC↗

arXiv2022

From an Authentication Question to a Public Social Event: Characterizing Birthday Sharing on Twitter

Date of birth (DOB) has historically been considered as private information and safe to use for authentication, but recent years have seen a shift towards wide public sharing. In this work we characterize how modern social media users are approaching the sharing of birthday wishes publicly online. Over 45 days, we collected over 2.8M tweets wishing happy birthday to 724K Twitter accounts. For 50K accounts, their age was likely mentioned revealing their DOB, and 10% were protected accounts. Our findings show that the majority of both public and protected accounts seem to be accepting of their birthdays and DOB being revealed online by their friends even when they do not have it listed on their profiles. We further complemented our findings through a survey to measure awareness of DOB disclosure issues and how people think about sharing different types of birthday-related information. Our analysis shows that giving birthday wishes to others online is considered a celebration and many users are quite comfortable with it. This view matches the trend also seen in security where the use of DOB in authentication process is no longer considered best practice.

cs.SI↗