arXiv subjects

Siyuan Ma

Publications and source records attributed to Siyuan Ma.

At least 55 records · Page 3Linked to original sources

arXiv2025

Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset

Machine unlearning has emerged as an effective strategy for forgetting specific information in the training data. However, with the increasing integration of visual data, privacy concerns in Vision Language Models (VLMs) remain underexplored. To address this, we introduce Facial Identity Unlearning Benchmark (FIUBench), a novel VLM unlearning benchmark designed to robustly evaluate the effectiveness of unlearning algorithms under the Right to be Forgotten setting. Specifically, we formulate the VLM unlearning task via constructing the Fictitious Facial Identity VQA dataset and apply a two-stage evaluation pipeline that is designed to precisely control the sources of information and their exposure levels. In terms of evaluation, since VLM supports various forms of ways to ask questions with the same semantic meaning, we also provide robust evaluation metrics including membership inference attacks and carefully designed adversarial privacy attacks to evaluate the performance of algorithms. Through the evaluation of four baseline VLM unlearning algorithms within FIUBench, we find that all methods remain limited in their unlearning performance, with significant trade-offs between model utility and forget quality. Furthermore, our findings also highlight the importance of privacy attacks for robust evaluations. We hope FIUBench will drive progress in developing more effective VLM unlearning algorithms.

cs.CV↗

arXiv2025

Epitaxial high-K AlBN barrier GaN HEMTs

We report a polarization-induced 2D electron gas (2DEG) at an epitaxial AlBN/GaN heterojunction grown on a SiC substrate. Using this 2DEG in a long conducting channel, we realize ultra-thin barrier AlBN/GaN high electron mobility transistors that exhibit current densities of more than 0.25 A/mm, clean current saturation, a low pinch-off voltage of -0.43 V, and a peak transconductance of 0.14 S/mm. Transistor performance in this preliminary realization is limited by the contact resistance. Capacitance-voltage measurements reveal that introducing 7 % B in the epitaxial AlBN barrier on GaN boosts the relative dielectric constant of AlBN to 16, higher than the AlN dielectric constant of 9. Epitaxial high-K barrier AlBN/GaN HEMTs can thus extend performance beyond the capabilities of current GaN transistors.

physics.app-ph↗

arXiv2024

Compositional Covariate Importance Testing via Partial Conjunction of Bivariate Hypotheses

Compositional data (i.e., data comprising random variables that sum up to a constant) arises in many applications including microbiome studies, chemical ecology, political science, and experimental designs. Yet when compositional data serve as covariates in a regression, the sum constraint renders every covariate automatically conditionally independent of the response given the other covariates, since each covariate is a deterministic function of the others. Since essentially all covariate importance tests and variable selection methods, including parametric ones, are at their core testing conditional independence, they are all completely powerless on regression problems with compositional covariates. In fact, compositionality causes ambiguity in the very notion of relevant covariates. To address this problem, we identify a natural way to translate the typical notion of relevant covariates to the setting with compositional covariates and establish that it is intuitive, well-defined, and unique. We then develop corresponding hypothesis tests and controlled variable selection procedures via a novel connection with \emph{bivariate} conditional independence testing and partial conjunction hypothesis testing. Finally, we provide theoretical guarantees of the validity of our methods, and through numerical experiments demonstrate that our methods are not only valid but also powerful across a range of data-generating scenarios.

stat.ME↗

arXiv2024

Fast training of large kernel models with delayed projections

Classical kernel machines have historically faced significant challenges in scaling to large datasets and model sizes--a key ingredient that has driven the success of neural networks. In this paper, we present a new methodology for building kernel machines that can scale efficiently with both data size and model size. Our algorithm introduces delayed projections to Preconditioned Stochastic Gradient Descent (PSGD) allowing the training of much larger models than was previously feasible, pushing the practical limits of kernel-based learning. We validate our algorithm, EigenPro4, across multiple datasets, demonstrating drastic training speed up over the existing methods while maintaining comparable or better classification accuracy.

stat.ML↗

arXiv2024

JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks

With the rapid advancements in Multimodal Large Language Models (MLLMs), securing these models against malicious inputs while aligning them with human values has emerged as a critical challenge. In this paper, we investigate an important and unexplored question of whether techniques that successfully jailbreak Large Language Models (LLMs) can be equally effective in jailbreaking MLLMs. To explore this issue, we introduce JailBreakV-28K, a pioneering benchmark designed to assess the transferability of LLM jailbreak techniques to MLLMs, thereby evaluating the robustness of MLLMs against diverse jailbreak attacks. Utilizing a dataset of 2, 000 malicious queries that is also proposed in this paper, we generate 20, 000 text-based jailbreak prompts using advanced jailbreak attacks on LLMs, alongside 8, 000 image-based jailbreak inputs from recent MLLMs jailbreak attacks, our comprehensive dataset includes 28, 000 test cases across a spectrum of adversarial scenarios. Our evaluation of 10 open-source MLLMs reveals a notably high Attack Success Rate (ASR) for attacks transferred from LLMs, highlighting a critical vulnerability in MLLMs that stems from their text-processing capabilities. Our findings underscore the urgent need for future research to address alignment vulnerabilities in MLLMs from both textual and visual inputs.

cs.CR↗

arXiv2024

Recent advances in deep learning and language models for studying the microbiome

Recent advancements in deep learning, particularly large language models (LLMs), made a significant impact on how researchers study microbiome and metagenomics data. Microbial protein and genomic sequences, like natural languages, form a language of life, enabling the adoption of LLMs to extract useful insights from complex microbial ecologies. In this paper, we review applications of deep learning and language models in analyzing microbiome and metagenomics data. We focus on problem formulations, necessary datasets, and the integration of language modeling techniques. We provide an extensive overview of protein/genomic language modeling and their contributions to microbiome studies. We also discuss applications such as novel viromics language modeling, biosynthetic gene cluster prediction, and knowledge integration for metagenomics studies.

q-bio.QM↗

arXiv2024

Toward satisfactory public accessibility: A crowdsourcing approach through online reviews to inclusive urban design

As urban populations grow, the need for accessible urban design has become urgent. Traditional survey methods for assessing public perceptions of accessibility are often limited in scope. Crowdsourcing via online reviews offers a valuable alternative to understanding public perceptions, and advancements in large language models can facilitate their use. This study uses Google Maps reviews across the United States and fine-tunes Llama 3 model with the Low-Rank Adaptation technique to analyze public sentiment on accessibility. At the POI level, most categories -- restaurants, retail, hotels, and healthcare -- show negative sentiments. Socio-spatial analysis reveals that areas with higher proportions of white residents and greater socioeconomic status report more positive sentiment, while areas with more elderly, highly-educated residents exhibit more negative sentiment. Interestingly, no clear link is found between the presence of disabilities and public sentiments. Overall, this study highlights the potential of crowdsourcing for identifying accessibility challenges and providing insights for urban planners.

cs.SI↗

arXiv2024

Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character

With the advent and widespread deployment of Multimodal Large Language Models (MLLMs), ensuring their safety has become increasingly critical. To achieve this objective, it requires us to proactively discover the vulnerability of MLLMs by exploring the attack methods. Thus, structure-based jailbreak attacks, where harmful semantic content is embedded within images, have been proposed to mislead the models. However, previous structure-based jailbreak methods mainly focus on transforming the format of malicious queries, such as converting harmful content into images through typography, which lacks sufficient jailbreak effectiveness and generalizability. To address these limitations, we first introduce the concept of "Role-play" into MLLM jailbreak attacks and propose a novel and effective method called Visual Role-play (VRP). Specifically, VRP leverages Large Language Models to generate detailed descriptions of high-risk characters and create corresponding images based on the descriptions. When paired with benign role-play instruction texts, these high-risk character images effectively mislead MLLMs into generating malicious responses by enacting characters with negative attributes. We further extend our VRP method into a universal setup to demonstrate its generalizability. Extensive experiments on popular benchmarks show that VRP outperforms the strongest baseline, Query relevant and FigStep, by an average Attack Success Rate (ASR) margin of 14.3% across all models.

cs.CR↗

arXiv2024

A scoping review of using Large Language Models (LLMs) to investigate Electronic Health Records (EHRs)

Electronic Health Records (EHRs) play an important role in the healthcare system. However, their complexity and vast volume pose significant challenges to data interpretation and analysis. Recent advancements in Artificial Intelligence (AI), particularly the development of Large Language Models (LLMs), open up new opportunities for researchers in this domain. Although prior studies have demonstrated their potential in language understanding and processing in the context of EHRs, a comprehensive scoping review is lacking. This study aims to bridge this research gap by conducting a scoping review based on 329 related papers collected from OpenAlex. We first performed a bibliometric analysis to examine paper trends, model applications, and collaboration networks. Next, we manually reviewed and categorized each paper into one of the seven identified topics: named entity recognition, information extraction, text similarity, text summarization, text classification, dialogue system, and diagnosis and prediction. For each topic, we discussed the unique capabilities of LLMs, such as their ability to understand context, capture semantic relations, and generate human-like text. Finally, we highlighted several implications for researchers from the perspectives of data resources, prompt engineering, fine-tuning, performance measures, and ethical concerns. In conclusion, this study provides valuable insights into the potential of LLMs to transform EHR research and discusses their applications and ethical considerations.

cs.ET↗

arXiv2023

PIMSAB: A Processing-In-Memory System with Spatially-Aware Communication and Bit-Serial-Aware Computation

Bit-serial Processing-In-Memory (PIM) is an attractive paradigm for accelerator architectures, for parallel workloads such as Deep Learning (DL), because of its capability to achieve massive data parallelism at a low area overhead and provide orders-of-magnitude data movement savings by moving computational resources closer to the data. While many PIM architectures have been proposed, improvements are needed in communicating intermediate results to consumer kernels, for communication between tiles at scale, for reduction operations, and for efficiently performing bit-serial operations with constants. We present PIMSAB, a scalable architecture that provides spatially aware communication network for efficient intra-tile and inter-tile data movement and provides efficient computation support for generally inefficient bit-serial compute patterns. Our architecture consists of a massive hierarchical array of compute-enabled SRAMs (CRAMs) and is codesigned with a compiler to achieve high utilization. The key novelties of our architecture are: (1) providing efficient support for spatially-aware communication by providing local H-tree network for reductions, by adding explicit hardware for shuffling operands, and by deploying systolic broadcasting, and (2) taking advantage of the divisible nature of bit-serial computations through adaptive precision, bit-slicing and efficient handling of constant operations. When compared against a similarly provisioned modern Tensor Core GPU (NVIDIA A100), across common DL kernels and an end-to-end DL network (Resnet18), PIMSAB outperforms the GPU by 3x, and reduces energy by 4.2x. We compare PIMSAB with similarly provisioned state-of-the-art SRAM PIM (Duality Cache) and DRAM PIM (SIMDRAM) and observe a speedup of 3.7x and 3.88x respectively.

cs.AR↗

arXiv2023

Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models

This paper presents a comprehensive survey of ChatGPT-related (GPT-3.5 and GPT-4) research, state-of-the-art large language models (LLM) from the GPT series, and their prospective applications across diverse domains. Indeed, key innovations such as large-scale pre-training that captures knowledge across the entire world wide web, instruction fine-tuning and Reinforcement Learning from Human Feedback (RLHF) have played significant roles in enhancing LLMs' adaptability and performance. We performed an in-depth analysis of 194 relevant papers on arXiv, encompassing trend analysis, word cloud representation, and distribution analysis across various application domains. The findings reveal a significant and increasing interest in ChatGPT-related research, predominantly centered on direct natural language processing applications, while also demonstrating considerable potential in areas ranging from education and history to mathematics, medicine, and physics. This study endeavors to furnish insights into ChatGPT's capabilities, potential implications, ethical concerns, and offer direction for future advancements in this field.

cs.CL↗

arXiv2023

Precise late-time asymptotics of scalar field in the interior of a subextreme Kerr black hole and its application in Strong Cosmic Censorship conjecture

In this work, we compute the precise late-time asymptotics for the scalar field in the interior of a non-static subextreme Kerr black hole, based on recent progress on deriving its precise asymptotics in the Kerr exterior region. This provides a new proof of the generic $H^1_{\text{loc}}$-inextendibility of the Kerr Cauchy horizon against scalar perturbations that is first shown by Luk--Sbierski (J. Func. Anal., 2016). The analogous results in Reissner--Nordström spacetimes are also discussed.

gr-qc↗

arXiv2023

Sharp decay for Teukolsky equation in Kerr spacetimes

In this work, we derive the global sharp decay, as both a lower and an upper bounds, for the spin $\pm \mathfrak{s}$ components, which are solutions to the Teukolsky equation, in the black hole exterior and on the event horizon of a slowly rotating Kerr spacetime. These estimates are generalized to any subextreme Kerr background under an integrated local energy decay estimate. Our results apply to the scalar field $(\mathfrak{s}=0)$, the Maxwell field $(\mathfrak{s}=1)$ and the linearized gravity $(\mathfrak{s}=2)$ and confirm the Price's law decay that is conjectured to be sharp. Our analyses rely on a novel global conservation law for the Teukolsky equation, and this new approach can be applied to derive the precise asymptotics for solutions to semilinear wave equations.

gr-qc↗

arXiv2023

Price's law for spin fields on a Schwarzschild background

In this work, we derive the globally precise late-time asymptotics for the spin-$\mathfrak{s}$ fields on a Schwarzschild background, including the scalar field $(\mathfrak{s}=0)$, the Maxwell field $(\mathfrak{s}=\pm 1)$ and the linearized gravity $(\mathfrak{s}=\pm 2)$. The conjectured Price's law in the physics literature which predicts the sharp rates of decay of the spin $s=\pm \mathfrak{s}$ components towards the future null infinity as well as in a compact region is shown. Further, we confirm the heuristic claim by Barack and Ori that the spin $+1, +2$ components have an extra power of decay at the event horizon than the conjectured Price's law. The asymptotics are derived via a unified, detailed analysis of the Teukolsky master equation that is satisfied by all these components.

math.AP↗

arXiv2022

Generically sharp decay for quasilinear wave equations with null condition

We are interested in the three-dimensional quasilinear wave equations with null condition. Global existence and pointwise decay for this model have been proved in the celebrated works of Klainerman \cite{Klainerman86} and Christodoulou \cite{Christodoulou86} for small smooth initial data. In this work, we illustrate the precise pointwise asymptotic behavior of the solutions for initial data posed on a hyperboloid and show that the decay rate $v^{-1}u^{-1}$ is optimal for a generic set of initial data.

math.AP↗

arXiv2022

Learning the joint distribution of two sequences using little or no paired data

We present a noisy channel generative model of two sequences, for example text and speech, which enables uncovering the association between the two modalities when limited paired data is available. To address the intractability of the exact model under a realistic data setup, we propose a variational inference approximation. To train this variational model with categorical data, we propose a KL encoder loss approach which has connections to the wake-sleep algorithm. Identifying the joint or conditional distributions by only observing unpaired samples from the marginals is only possible under certain conditions in the data distribution and we discuss under what type of conditional independence assumptions that might be achieved, which guides the architecture designs. Experimental results show that even tiny amount of paired data (5 minutes) is sufficient to learn to relate the two modalities (graphemes and phonemes here) when a massive amount of unpaired data is available, paving the path to adopting this principled approach for all seq2seq models in low data resource regimes.

cs.LG↗

arXiv2022

Analysis and design of transition radiation in layered uniaxial crystals using Tandem neural networks

With the flourishing development of nanophotonics, Cherenkov radiation pattern can be designed to achieve superior performance in particle detection by fine-tuning the properties of metamaterials such as photonic crystals (PCs) surrounding the swift particle. However, the radiation pattern can be sensitive to the geometry and material properties of PCs, such as periodicity, unit thickness, and dielectric fraction, making direct analysis and inverse design difficult. In this article, we propose a systematic method to analyze and design PC-based transition radiation, which is assisted by deep learning neural networks. By matching boundary conditions at the interfaces, Cherenkov-like radiation of multilayered structures can be resolved analytically using the cascading scattering matrix method, despite the optical axes not being aligned with the swift electron trajectory. Once well trained, forward deep learning neural networks can be utilized to predict the radiation pattern without further direct electromagnetic simulations; moreover, Tandem neural networks have been proposed to inversely design the geometry and/or material properties for desired Cherenkov radiation pattern. Our proposal demonstrates a promising strategy for dealing with layered-medium-based Cherenkov radiation detectors, and it can be extended for other emerging metamaterials, such as photonic time crystals.

physics.optics↗

arXiv2022

Nonlinear radiation gauge for near Kerr spacetimes

In this paper, we introduce and explore the properties of a new gauge choice for the vacuum Einstein equation inspired by the ingoing and outgoing radiation gauges (IRG, ORG) for the linearized vacuum Einstein equation introduced by Chrzanowski in his work on metric reconstruction [DOI:10.1103/PhysRevD.11.2042] on the Kerr background. It has been shown by Price, Shankar and Whiting [arXiv:gr-qc/0611070] that the IRG/ORG are consistent gauges for the linearized vacuum Einstein equation on Petrov type II backgrounds. In [arXiv:1903.03859], the ORG was used in proving linearized stability for the Kerr spacetime, and the new non-linear radiation gauge introduced here is a direct generalization of that gauge condition, and is intended to be used to study the stability of Kerr black holes under the evolution generated by the vacuum Einstein equation.

gr-qc↗