arXiv subjects

Zachary Liu

Publications and source records attributed to Zachary Liu.

4 recordsLinked to original sources

arXiv2026

Equitable System-Prompt Selection via Constrained Mixed-Strategy GroupDRO

Large language models are increasingly used for information seeking, yet semantically equivalent questions phrased in different ways can receive answers of considerably different quality. System prompts are widely employed to steer response behavior, but they are typically optimized for average-case quality, so some question phrasings may still receive incomplete or low-quality answers. To address this, we formulate a constrained mixed-strategy GroupDRO framework for system-prompt selection. Instead of optimizing the system-prompt text, the framework assigns weights to system prompts in an existing pool to minimize the worst-case information-quality loss across evaluation metrics and groups, while constraining the mean loss to stay close to that of average-based selection. Because pool generation and selection are decoupled, the method applies to any system-prompt pool and can leverage an ensemble of complementary system prompts rather than a single one. Across five LLMs on two bilingual medical and consumer-finance benchmarks, the constrained method reduces the Overall Mean, Worst 25% Mean, and Worst by 13.1%, 13.2%, and 13.7% on average relative to no mitigation while keeping overall quality close to Average selection. Its multi-prompt weights reveal complementarity across metric-group pairs. Code and data are available at https://github.com/Rainxu09/equitable-system-prompt-selection.

cs.CL↗

arXiv2025

OmniBench: Towards The Future of Universal Omni-Language Models

Recent advancements in multimodal large language models (MLLMs) have aimed to integrate and interpret data across diverse modalities. However, the capacity of these models to concurrently process and reason about multiple modalities remains underexplored, partly due to the lack of comprehensive modality-wise benchmarks. We introduce OmniBench, a novel benchmark designed to rigorously evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define language models capable of such tri-modal processing as the omni-language models (OLMs). OmniBench is distinguished by high-quality human annotations, ensuring that accurate responses require integrated understanding and reasoning across all three modalities. Our main findings reveal that: i) open-source OLMs exhibit critical limitations in instruction-following and reasoning capabilities within tri-modal contexts; and ii) most baselines models perform poorly (below 50% accuracy) even when provided with alternative textual representations of images or/and audio. These results suggest that the ability to construct a consistent context from text, image, and audio is often overlooked in existing MLLM training paradigms. To address this gap, we curate an instruction tuning dataset of 84.5K training samples, OmniInstruct, for training OLMs to adapt to tri-modal contexts. We advocate for future research to focus on developing more robust tri-modal integration techniques and training strategies to enhance OLMs. Codes, data and live leaderboard could be found at https://m-a-p.ai/OmniBench.

cs.CL↗

arXiv2025

Exercising the CCPA Opt-out Right on Android: Legally Mandated but Practically Challenging

Many mobile apps' business model is based on sharing user data with ad networks to deliver personalized ads. The California Consumer Privacy Act (CCPA) gives California residents a right to opt out. In two experiments we evaluate to which extent popular Android apps enable California residents to exercise their right. In our first experiment -- manually exercising the right via app-level UIs -- we find that only 48 out of 100 apps implement a respective setting, which suggests that CCPA opt-out right compliance on the Android platform is generally low. In our second experiment -- automatically exercising the opt-out right by sending Global Privacy Control (GPC) signals -- we find for an app dataset of 1,811 apps that GPC is largely ineffective. While we estimate with 95% confidence that 62%--81% of apps in our app dataset must respect the CCPA opt-out right, many apps do not do so. Our evaluation of disabling apps' access to the AdID -- which is technically not intended for exercising the CCPA opt-out right but could be practically effective -- does not change our conclusion. For example, when sending GPC signals and disabling apps' access to the AdID, 338 apps still had the ccpa status of the ad network Vungle set to opted_in while only 26 had set it to opted_out. Overall, our results suggest a compliance gap as California residents have no effective way of exercising their CCPA opt-out right on the Android platform; neither at the app nor at the platform-level. We think that re-purposing the Android AdID setting as an opt-out right setting with legal meaning under the CCPA and other laws could close this gap and improve users' privacy on the platform significantly.

cs.CR↗

arXiv2024

LIME: Less Is More for MLLM Evaluation

Multimodal Large Language Models (MLLMs) are evaluated on various benchmarks, such as image captioning, visual question answering, and reasoning. However, many of these benchmarks include overly simple or uninformative samples, complicating the effective distinction of different MLLMs' performance. Furthermore, evaluating models across numerous benchmarks incurs a significant computational burden. To address these issues, we propose LIME (Less Is More for MLLM Evaluation), a refined and efficient benchmark curated through a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that necessitate image-based understanding. Our experiments indicate that LIME reduces the number of samples by 76% and evaluation time by 77%, while also providing a more effective means of distinguishing the capabilities of different models. Notably, we find that traditional automatic metrics, such as CIDEr, are inadequate for assessing MLLMs' captioning performance; excluding the caption task score yields a more accurate reflection of overall model performance. All code and data are available at https://github.com/kangreen0210/LIME.

cs.CV↗