Searcharxiv⌕ Search

arXiv subjects

Ayaan Khan

Publications and source records attributed to Ayaan Khan.

3 recordsLinked to original sources

Calibrate, Don't Retrain: Decomposing the Value of Annotator Disagreement in Cyberbullying Severity

Machine learning models are often trained on labeled data produced by multiple human annotators. When those annotators disagree, their labels are commonly collapsed into a single majority vote, which discards information about the underlying disagreement. We examine the cost of this practice for cyberbullying severity, a subjective judgment on which annotators disagree heavily. In the corpus we study, all 5 annotators assigned the same severity label to only 4.1% of comments. Turning 5 judgments into one label hides this disagreement and obscures how much the annotators differed in their assessments. In this paper, we ask whether a model can preserve this information by predicting the distribution of annotator judgments instead of a single majority label. We find 3 main results. First, accuracy against the majority label cannot tell competent models apart: 2 models with similar accuracy can differ greatly in how well they match the distribution of annotator judgments. Second, predicting the full distribution requires little extra effort. Calibrating an existing model, which adjusts its output probabilities after training, captures most of the improvement without retraining, across 3 datasets covering severity, abuse, and irony, while training on label distributions adds a small further gain. Third, we examine a dataset that includes annotator demographics, where a single shared model has one advantage and one limit. The advantage is that a model predicting the distribution of judgments matches each demographic group's judgments significantly better than a model trained on majority labels, so both groups gain. The limit is that it does not narrow the gap in performance between the groups, because one shared model cannot encode which group is right on a given comment. Preserving annotator disagreement is therefore a low-cost step toward fairer moderation, though not a complete one.

cs.SI↗

Domain Agnostic Text Redaction from Natural Language Rules using Instruction Tuning

With the increasing digitization of personal and corporate communication, the automatic sanitization of textual data has become a crucial component of data privacy and compliance frameworks. Traditional text sanitization solutions are majorly suitable for obscuring sensitive data with standard structure such as Personal Identifiable Information (PII). These solutions do not provide transparent justification for their redaction, which makes it difficult to audit them. This paper introduces an explainable, domain-agnostic text redaction solution that uses natural language rules of redaction, applied via an instruction-tuned language model, to identify and redact sensitive information in unstructured documents. Unlike traditional text sanitization, this method enables a user to conveniently define any sensitive information; which may be structured (e.g.\ PII) or unstructured (e.g.\ legal terms and conditions) in natural language. A general-purpose LLM generates or augments these natural language rules of redaction from the user's definition, which are then used to instruction-fine-tune a smaller language model that reasons the rules step-by-step over any given document to identify and redact the corresponding sensitive content, while providing transparent justifications for each redaction and highlighting the specific rule that triggered the decision. This explanation is generated in natural language to support human reviewers and auditors in understanding why specific content was redacted. A reconstruction-based metric is used to estimate the probability of recovering redacted information from the sanitized document, quantifying redaction coverage. The solution shows high reconstruction error and high redaction precision, making it suitable for automated text sanitization in critical applications such as legal discovery, medical documentation, and corporate information governance.

cs.CL↗

SPILLOVER: Measuring Cyberbullying NormPropagation on Social Media

While certain aspects of cyberbullying (CB) such as its factors and prevalence have been studied extensively, relatively little attention has been given to specifically how the aggression transfers from comment to comment. This understanding could have important implications for designing better anti-bullying features. In this paper, we study multiple aspects of the nature of this aggression transference in social media sessions. Using data from 32,754 consecutive comment pairs from 430 Instagram sessions, we find that a preceding CB comment substantially raises the odds of the next comment being CB, an effect confirmed by session fixed-effects controls and driven primarily by cross-user spread. We also find that $\text{CB} \to \text{CB}$ pairs are more textually similar than $\text{NoCB} \to \text{CB}$ pairs across five complementary methods, and that this pattern holds under a matched cross-session baseline that rules out shared vocabulary, session toxicity, and session length as confounds. Moreover, non-aggressive replies grow more negative as preceding CB severity increases, a graded pattern consistent with automatic emotional influence below the threshold of overt aggression. These key findings replicate across three independent datasets (Reddit, Wikipedia Detox, and SOCC), with spillover rates that track platform visibility design. Finally, we show that a single binary feature (whether the prior comment was CB) improves prediction over session-level baselines and over a fine-tuned HateBERT classifier, serving as a real-time moderation signal that targets the spreading chain rather than individual offenders.

cs.SI↗