arXiv subjects

Pirooz Shamsinejadbabaki

Publications and source records attributed to Pirooz Shamsinejadbabaki.

3 recordsLinked to original sources

arXiv2026

QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI

The rapid progress of generative artificial intelligence has exposed fundamental limitations in existing evaluation methodologies, particularly for open-ended, creative, and human-facing tasks. Traditional automatic metrics rely on surface-level statistical similarity and often fail to reflect human perceptions of quality, while purely human evaluation, although reliable, is costly, subjective, and difficult to scale. Recent approaches using large language models as evaluators offer improved scalability but frequently lack explicit grounding in human-defined evaluation principles, leading to bias and inconsistency. In this paper, we introduce Quantifying Qualitative Judgment (QQJ), a scalable and human-centric evaluation framework that explicitly bridges the gap between human judgment and automated assessment. QQJ separates the definition of quality from its execution by anchoring evaluation in expert-designed, multi-dimensional rubrics and calibrating large language model evaluators to align with expert reasoning using a small, high-quality annotation set. This design enables consistent, interpretable, and scalable evaluation across diverse generative tasks and modalities. Extensive experiments on text and image generation demonstrate that QQJ achieves substantially stronger alignment with human judgment than traditional automatic metrics and unconstrained LLM-based evaluators. Moreover, QQJ exhibits improved stability across repeated evaluations and superior diagnostic capability in identifying critical failure modes such as hallucination and intent mismatch. These results indicate that structured qualitative judgment can be operationalized at scale without sacrificing interpretability or human alignment, positioning QQJ as a practical foundation for reliable evaluation of modern generative AI systems.

cs.AI↗

arXiv2026

Res-MIA: A Training-Free Resolution-Based Membership Inference Attack on Federated Learning Models

Membership inference attacks (MIAs) pose a serious threat to the privacy of machine learning models by allowing adversaries to determine whether a specific data sample was included in the training set. Although federated learning (FL) is widely regarded as a privacy-aware training paradigm due to its decentralized nature, recent evidence shows that the final global model can still leak sensitive membership information through black-box access. In this paper, we introduce Res-MIA, a novel training-free and black-box membership inference attack that exploits the sensitivity of deep models to high-frequency input details. Res-MIA progressively degrades the input resolution using controlled downsampling and restoration operations, and analyzes the resulting confidence decay in the model's predictions. Our key insight is that training samples exhibit a significantly steeper confidence decline under resolution erosion compared to non-member samples, revealing a robust membership signal. Res-MIA requires no shadow models, no auxiliary data, and only a limited number of forward queries to the target model. We evaluate the proposed attack on a federated ResNet-18 trained on CIFAR-10, where it consistently outperforms existing training-free baselines and achieves an AUC of up to 0.88 with minimal computational overhead. These findings highlight frequency-sensitive overfitting as an important and previously underexplored source of privacy leakage in federated learning, and emphasize the need for privacy-aware model designs that reduce reliance on fine-grained, non-robust input features.

cs.CR↗

arXiv2022

Deep Learning Methods for Software Requirement Classification: A Performance Study on the PURE dataset

Requirement engineering (RE) is the first and the most important step in software production and development. The RE is aimed to specify software requirements. One of the tasks in RE is the categorization of software requirements as functional and non-functional requirements. The functional requirements (FR) show the responsibilities of the system while non-functional requirements represent the quality factors of software. Discrimination between FR and NFR is a challenging task. Nowadays Deep Learning (DL) has entered all fields of engineering and has increased accuracy and reduced time in their implementation process. In this paper, we use deep learning for the classification of software requirements. Five prominent DL algorithms are trained for classifying requirements. Also, two voting classification algorithms are utilized for creating ensemble classifiers based on five DL methods. The PURE, a repository of Software Requirement Specification (SRS) documents, is selected for our experiments. We created a dataset from PURE which contains 4661 requirements where 2617 requirements are functional and the remaining are non-functional. Our methods are applied to the dataset and their performance analysis is reported. The results show that the performance of deep learning models is satisfactory and the voting mechanisms provide better results.

cs.SE↗