arXiv subjects

Zhaoyi Li

Publications and source records attributed to Zhaoyi Li.

At least 37 records · Page 2Linked to original sources

arXiv2026

Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting

Black-box adversarial attacks on Large Vision-Language Models (LVLMs) are challenging due to missing gradients and complex multimodal boundaries. While prior state-of-the-art transfer-based approaches like M-Attack perform well using local crop-level matching between source and target images, we find this induces high-variance, nearly orthogonal gradients across iterations, violating coherent local alignment and destabilizing optimization. We attribute this to (i) ViT translation sensitivity that yields spike-like gradients and (ii) structural asymmetry between source and target crops. We reformulate local matching as an asymmetric expectation over source transformations and target semantics, and build a gradient-denoising upgrade to M-Attack. On the source side, Multi-Crop Alignment (MCA) averages gradients from multiple independently sampled local views per iteration to reduce variance. On the target side, Auxiliary Target Alignment (ATA) replaces aggressive target augmentation with a small auxiliary set from a semantically correlated distribution, producing a smoother, lower-variance target manifold. We further reinterpret momentum as Patch Momentum, replaying historical crop gradients; combined with a refined patch-size ensemble (PE+), this strengthens transferable directions. Together these modules form M-Attack-V2, a simple, modular enhancement over M-Attack that substantially improves transfer-based black-box attacks on frontier LVLMs: boosting success rates on Claude-4.0 from 8% to 30%, Gemini-2.5-Pro from 83% to 97%, and GPT-5 from 98% to 100%, outperforming prior black-box LVLM attacks. Code and data are publicly available at: https://github.com/vila-lab/M-Attack-V2.

cs.LG↗

arXiv2026

Dataset Distillation via Committee Voting

Dataset distillation aims to synthesize a compact yet representative dataset that preserves the essential characteristics of the original data for efficient model training. Existing methods mainly focus on improving data-synthetic alignment or scaling distillation to large datasets. In this work, we propose $\textbf{C}$ommittee $\textbf{V}$oting for $\textbf{D}$ataset $\textbf{D}$istillation ($\textbf{CV-DD}$), an orthogonal approach that leverages the collective knowledge of multiple models to produce higher-quality distilled data. We first establish a strong baseline that achieves state-of-the-art performance through modern architectural and optimization choices. By integrating distributions and predictions from multiple models and generating high-quality soft labels, our method captures a broader range of data characteristics, reduces model-specific bias and the impact of distribution shifts, and significantly improves generalization. This voting-based strategy enhances diversity and robustness, alleviates overfitting, and improves post-evaluation performance. Extensive experiments across multiple datasets and IPC settings demonstrate that CV-DD consistently outperforms single- and multi-model distillation methods and generalizes well to non-training-based frameworks and challenging synthetic-to-real transfer tasks. Code is available at: https://github.com/Jiacheng8/CV-DD.

cs.CV↗

arXiv2026

Fine-Tuning Flow Matching via Maximum Likelihood Estimation of Reconstructions

Flow Matching (FM) models achieve remarkable results in generative tasks. Building upon diffusion models, FM's simulation-free training paradigm enables simplicity and efficiency but introduces a train-inference gap: model outputs cannot be assessed during training. Moreover, the straight flow assumption suffers from some inherent limitations. To address this, we propose to fine-tune FM via Maximum Likelihood Estimation (MLE) of reconstructions -- enabled by FM's smooth ODE formulation, unlike the stochastic differential equations (SDEs) in diffusion models. We first theoretically analyze the relationship between training loss and inference error in FM under numerical precision constraints. We then propose an easy-to-implement fine-tuning framework based on MLE of reconstructions, with flexibility for sophisticated extensions. Building on this, we incorporate a generalized artificial viscosity term that enhances flow stability and robustness, accompanied by a direct parameterization method and rigorous theoretical guarantees. Experiments demonstrate our method's effectiveness across diverse settings: a toy example provides mechanistic insights into the fine-tuning process, while large-scale evaluations on meteorological forecasting and robotic manipulation policies validate reliable performance improvements.

cs.LG↗

arXiv2025

Optimal Quantum Purity Amplification

Quantum purity amplification (QPA) provides a novel approach to counteracting the pervasive noise that degrades quantum states. We present the optimal QPA protocol for general quantum systems and global noise, resolving a two-decade open problem. Under strong depolarization, our protocol achieves an exponential reduction in sample complexity over the best-known methods. We provide an efficient implementation of the protocol based on generalized quantum phase estimation. Additionally, we introduce SWAPNET, a sparse and shallow circuit that enables QPA for near-term experiments. Simulations in both digital and analog quantum settings, along with experiments on superconducting quantum processors, confirm the protocol's robustness and practical utility. Our findings suggest that QPA could improve the performance of quantum information processing tasks, particularly in the context of Noisy Intermediate-Scale Quantum (NISQ) devices, where reducing the effect of noise with limited resources is critical.

quant-ph↗

arXiv2025

Wide field-of-view and large depth-of-field metalenses

The ability to visualize both macroscopic and microscopic features over an extended field of view is essential for endoscopic imaging and other applications ranging from machine vision to microscopy. However, miniaturizing endoscopes introduces inherent trade-offs between size and optical performance, including field-of-view (FOV), depth-of-field (DOF), and resolution. These constraints limit the use of microendoscopes in clinical settings such as early cancer detection within narrow, hard-to-access anatomical regions, including the lung, ovaries, and pancreas. State-of-the-art microendoscopes typically rely on microlens assemblies that increase both cost and size. Their large f-numbers also hinder the collection of high-resolution information from live tissue. In this work, we present two compact metalens designs that provide wide FOV, extended DOF, and high resolution, enabled by custom-tailored point spread functions (PSFs). The devices achieve a full 172 degrees FOV, an extended DOF from 0.4 mm to beyond 300 mm, and a resolution of 30 line pairs per millimeter, all within a 1 mm x 1 mm x 0.2 mm footprint. A key advantage of our approach is the ability to transition seamlessly between low and high magnification without mechanical refocusing. Final images are reconstructed through backend deconvolution, highlighting the potential of hybrid imaging systems that integrate computational techniques with flat-optics components.

physics.optics↗

arXiv2025

Wafer-scale conformal metasurface optics

Curved and conformal optics offer significant advantages by unlocking additional geometric degrees of freedom for optical design. These capabilities enable enhanced optical performance and are essential for meeting non-optical constraints, such as those imposed by ergonomics, aerodynamics, or wearability. However, existing fabrication techniques such as direct electron or laser beam writing on curved substrates, and soft-stamp-based transfer or nanoimprint lithography suffer from limitations in scalability, yield, geometry control, and alignment accuracy. Here, we present a scalable fabrication strategy for curved and conformal metasurface optics leveraging thermoforming, an industry-standard, high-throughput manufacturing process widely used for shaping thermoplastics. Our approach uniquely enables wafer-scale production of highly curved metasurface optics, achieving sub-millimeter radii of curvature and micron-level alignment precision. To guide the design and fabrication process, we developed a thermorheological model that accurately predicts and compensates for the large strains induced during thermoforming. This allows for precise control of metasurface geometry and preservation of optical function, yielding devices with diffraction-limited performance. As a demonstration, we implemented an artificial compound eye comprising freeform micro-metalens arrays. Compared to traditional micro-optical counterparts, the device exhibits an expanded field of view, reduced aberrations, and improved uniformity, highlighting the potential of thermoformed metasurfaces for next-generation optical systems.

physics.optics↗

arXiv2025

Tunable Multistage Refrigeration via Geometrically Frustrated Triangular Lattice Antiferromagnet for Space Cooling

Low-temperature refrigeration technology constitutes a crucial component in space exploration. The small-scale, low-vibration Stirling-type pulse tube refrigerators hold significant application potential for space cooling. However, the efficient operation of current Stirling-type pulse tube cryocoolers in space cooling applications remains challenging due to the rapid decay of the heat capacity of regenerative materials below 10 K. This study adopts a novel material strategy: using a novel high-spin S = 7/2 magnetic regenerative material, Gd2O2Se, we construct a multistage tunable regenerative material structure to achieve an efficient cooling approach to the liquid helium temperature range. Under substantial geometric frustration from a double-layered triangular lattice, it exhibits two-step specific heat transition peaks at 6.22 K and 2.11 K, respectively. Its ultrahigh specific heat and broad two-step transition temperature range effectively bridge the gap between commercially used high-heat-capacity materials. Experimental verification shows that when Gd2O2Se is combined with Er3Ni and HoCu2 in the Stirling-type pulse tube cryocooler, the cooling efficiency of the pulse tube increases by 66.5 % at 7 K, and the minimum achievable temperature reaches 5.85 K. These results indicate that Gd2O2Se is an ideal magnetic regenerative material for space cooling

cond-mat.mtrl-sci↗

arXiv2025

A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1

Despite promising performance on open-source large vision-language models (LVLMs), transfer-based targeted attacks often fail against closed-source commercial LVLMs. Analyzing failed adversarial perturbations reveals that the learned perturbations typically originate from a uniform distribution and lack clear semantic details, resulting in unintended responses. This critical absence of semantic information leads commercial black-box LVLMs to either ignore the perturbation entirely or misinterpret its embedded semantics, thereby causing the attack to fail. To overcome these issues, we propose to refine semantic clarity by encoding explicit semantic details within local regions, thus ensuring the capture of finer-grained features and inter-model transferability, and by concentrating modifications on semantically rich areas rather than applying them uniformly. To achieve this, we propose a simple yet highly effective baseline: at each optimization step, the adversarial image is cropped randomly by a controlled aspect ratio and scale, resized, and then aligned with the target image in the embedding space. While the naive source-target matching method has been utilized before in the literature, we are the first to provide a tight analysis, which establishes a close connection between perturbation optimization and semantics. Experimental results confirm our hypothesis. Our adversarial examples crafted with local-aggregated perturbations focused on crucial regions exhibit surprisingly good transferability to commercial LVLMs, including GPT-4.5, GPT-4o, Gemini-2.0-flash, Claude-3.5/3.7-sonnet, and even reasoning models like o1, Claude-3.7-thinking and Gemini-2.0-flash-thinking. Our approach achieves success rates exceeding 90% on GPT-4.5, 4o, and o1, significantly outperforming all prior state-of-the-art attack methods with lower $\ell_1/\ell_2$ perturbations.

cs.CV↗

arXiv2025

A Unified Frequency Domain Decomposition Framework for Interpretable and Robust Time Series Forecasting

Current approaches for time series forecasting, whether in the time or frequency domain, predominantly use deep learning models based on linear layers or transformers. They often encode time series data in a black-box manner and rely on trial-and-error optimization solely based on forecasting performance, leading to limited interpretability and theoretical understanding. Furthermore, the dynamics in data distribution over time and frequency domains pose a critical challenge to accurate forecasting. We propose FIRE, a unified frequency domain decomposition framework that provides a mathematical abstraction for diverse types of time series, so as to achieve interpretable and robust time series forecasting. FIRE introduces several key innovations: (i) independent modeling of amplitude and phase components, (ii) adaptive learning of weights of frequency basis components, (iii) a targeted loss function, and (iv) a novel training paradigm for sparse data. Extensive experiments demonstrate that FIRE consistently outperforms state-of-the-art models on long-term forecasting benchmarks, achieving superior predictive performance and significantly enhancing interpretability of time series

cs.LG↗

arXiv2025

Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents

CAPTCHAs have been a critical bottleneck for deploying web agents in real-world applications, often blocking them from completing end-to-end automation tasks. While modern multimodal LLM agents have demonstrated impressive performance in static perception tasks, their ability to handle interactive, multi-step reasoning challenges like CAPTCHAs is largely untested. To address this gap, we introduce Open CaptchaWorld, the first web-based benchmark and platform specifically designed to evaluate the visual reasoning and interaction capabilities of MLLM-powered agents through diverse and dynamic CAPTCHA puzzles. Our benchmark spans 20 modern CAPTCHA types, totaling 225 CAPTCHAs, annotated with a new metric we propose: CAPTCHA Reasoning Depth, which quantifies the number of cognitive and motor steps required to solve each puzzle. Experimental results show that humans consistently achieve near-perfect scores, state-of-the-art MLLM agents struggle significantly, with success rates at most 40.0% by Browser-Use Openai-o3, far below human-level performance, 93.3%. This highlights Open CaptchaWorld as a vital benchmark for diagnosing the limits of current multimodal agents and guiding the development of more robust multimodal reasoning systems. Code and Data are available at this https URL.

cs.AI↗

arXiv2025

What Makes a Good Reasoning Chain? Uncovering Structural Patterns in Long Chain-of-Thought Reasoning

Recent advances in reasoning with large language models (LLMs) have popularized Long Chain-of-Thought (LCoT), a strategy that encourages deliberate and step-by-step reasoning before producing a final answer. While LCoTs have enabled expert-level performance in complex tasks, how the internal structures of their reasoning chains drive, or even predict, the correctness of final answers remains a critical yet underexplored question. In this work, we present LCoT2Tree, an automated framework that converts sequential LCoTs into hierarchical tree structures and thus enables deeper structural analysis of LLM reasoning. Using graph neural networks (GNNs), we reveal that structural patterns extracted by LCoT2Tree, including exploration, backtracking, and verification, serve as stronger predictors of final performance across a wide range of tasks and models. Leveraging an explainability technique, we further identify critical thought patterns such as over-branching that account for failures. Beyond diagnostic insights, the structural patterns by LCoT2Tree support practical applications, including improving Best-of-N decoding effectiveness. Overall, our results underscore the critical role of internal structures of reasoning chains, positioning LCoT2Tree as a powerful tool for diagnosing, interpreting, and improving reasoning in LLMs.

cs.AI↗

arXiv2025

Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning

Catastrophic forgetting (CF) poses a significant challenge in machine learning, where a model forgets previously learned information upon learning new tasks. Despite the advanced capabilities of Large Language Models (LLMs), they continue to face challenges with CF during continual learning. The majority of existing research focuses on analyzing forgetting patterns through a singular training sequence, thereby overlooking the intricate effects that diverse tasks have on model behavior. Our study explores CF across various settings, discovering that model forgetting is influenced by both the specific training tasks and the models themselves. To this end, we interpret forgetting by examining the function vector (FV), a compact representation of functions in LLMs, offering a model-dependent indicator for the occurrence of CF. Through theoretical and empirical analyses, we demonstrated that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions. Leveraging these insights, we propose a novel function vector guided training methodology, incorporating a regularization technique to stabilize the FV and mitigate forgetting. Empirical tests on four benchmarks confirm the effectiveness of our proposed training method, substantiating our theoretical framework concerning CF and model function dynamics. We plan to make our code publicly accessible in the near future.

cs.LG↗

arXiv2025

ICODE: Modeling Dynamical Systems with Extrinsic Input Information

Learning models of dynamical systems with external inputs, which may be, for example, nonsmooth or piecewise, is crucial for studying complex phenomena and predicting future state evolution, which is essential for applications such as safety guarantees and decision-making. In this work, we introduce \emph{Input Concomitant Neural ODEs (ICODEs)}, which incorporate precise real-time input information into the learning process of the models, rather than treating the inputs as hidden parameters to be learned. The sufficient conditions to ensure the model's contraction property are provided to guarantee that system trajectories of the trained model converge to a fixed point, regardless of initial conditions across different training processes. We validate our method through experiments on several representative real dynamics: Single-link robot, DC-to-DC converter, motion dynamics of a rigid body, Rabinovich-Fabrikant equation, Glycolytic-glycogenolytic pathway model, and heat conduction equation. The experimental results demonstrate that our proposed ICODEs efficiently learn the ground truth systems, achieving superior prediction performance under both typical and atypical inputs. This work offers a valuable class of neural ODE models for understanding physical systems with explicit external input information, with potentially promising applications in fields such as physics and robotics. Our code is available online at https://github.com/EEE-ai59/ICODE.git.

cs.LG↗

arXiv2025

Quantum Fluctuation-enhanced Milli-Kelvin Magnetic Refrigeration in Triangular Lattice Magnet GdBO3

Rare-earth-based triangular lattice antiferromagnets, with strong quantum fluctuations and weak magnetic interactions, can often retain large magnetic entropy down to very low temperatures, making them excellent candidates for magnetic refrigeration at ultra-low temperatures. These materials exhibit a substantial magnetocaloric effect (MCE) due to enhanced spin fluctuations, particularly near quantum critical points, which leads to significant changes in magnetic entropy. This study reports on the crystal growth, structure, magnetism, and MCE of a Gd-based triangular lattice material, GdBO3, characterized by a large spin quantum number (S = 7/2). Successive phase transitions (T1 = 0.52 K, T2 = 0.88 K, and T3 = 1.77 K) were observed in zero-field specific heat measurements. Furthermore, thermal dynamic analysis under external magnetic fields identified five distinct phase regions and three quantum critical points for GdBO3. Due to its broad specific heat features and the high density of magnetic Gd3+ ions, we achieved a minimum temperature of 50 mK near the field-induced quantum critical point, using a custom-designed GdBO3-based adiabatic demagnetization refrigerator. Our findings reveal significant quantum fluctuations below 2 K, demonstrating GdBO3's potential for milli-Kelvin magnetic cooling applications.

cond-mat.str-el↗

arXiv2025

Learning to Substitute Components for Compositional Generalization

Despite the rising prevalence of neural language models, recent empirical evidence suggests their deficiency in compositional generalization. One of the current de-facto solutions to this problem is compositional data augmentation, which aims to introduce additional compositional inductive bias. However, existing handcrafted augmentation strategies offer limited improvement when systematic generalization of neural language models requires multi-grained compositional bias (i.e., not limited to either lexical or structural biases alone) or when training sentences have an imbalanced difficulty distribution. To address these challenges, we first propose a novel compositional augmentation strategy called Component Substitution (CompSub), which enables multi-grained composition of substantial substructures across the entire training set. Furthermore, we introduce the Learning Component Substitution (LCS) framework. This framework empowers the learning of component substitution probabilities in CompSub in an end-to-end manner by maximizing the loss of neural language models, thereby prioritizing challenging compositions with elusive concepts and novel contexts. We extend the key ideas of CompSub and LCS to the recently emerging in-context learning scenarios of pre-trained large language models (LLMs), proposing the LCS-ICL algorithm to enhance the few-shot compositional generalization of state-of-the-art (SOTA) LLMs. Theoretically, we provide insights into why applying our algorithms to language models can improve compositional generalization performance. Empirically, our results on four standard compositional generalization benchmarks(SCAN, COGS, GeoQuery, and COGS-QL) demonstrate the superiority of CompSub, LCS, and LCS-ICL, with improvements of up to 66.5%, 10.3%, 1.4%, and 8.8%, respectively.

cs.CL↗

arXiv2025

Kagome Metal GdNb$_6$Sn$_6$: A 4d Playground for Topological Magnetism and Electron Correlations

Magnetic kagome metals have garnered considerable attention as an ideal platform for investigating intrinsic topological structures, frustrated magnetism, and electron correlation effects. In this work, we present the synthesis and detailed characterization of GdNb$_6$Sn$_6$, a metal that features a niobium-based kagome lattice and a frustrated triangular gadolinium network. The compound adopts the HfFe$_6$Ge$_6$-type crystal structure, with lattice parameters of a = b = 5.765(4) Å and c = 9.536(8) Å. Magnetic susceptibility and specific heat measurements reveal a magnetic transition near 2.3 K. Electrical transport data confirm metallic behavior, unsaturated positive magnetoresistance, and a hole-dominated multiband Hall effect. Furthermore, first-principles calculations indicate that Nb-4d orbitals predominantly contribute to the electronic states near the Fermi energy, with the band structure showing multiple topologically nontrivial crossings around the Fermi surface. This study also compares GdNb$_6$Sn$_6$ with GdV$_6$Sn$_6$, highlighting their similarities and differences. Our findings pave the way for exploring RNb$_6$Sn$_6$ (R = rare earth) with customized substitutions of R sites to fine-tune their properties.

cond-mat.mtrl-sci↗

arXiv2024

Heterogeneous Freeform Metasurfaces: A Platform for Advanced Broadband Dispersion Engineering

Metasurfaces, with their ability to control electromagnetic waves, hold immense potential in optical device design, especially for applications requiring precise control over dispersion. This work introduces an approach to dispersion engineering using heterogeneous freeform metasurfaces, which overcomes the limitations of conventional metasurfaces that often suffer from poor transmission, narrow bandwidth, and restricted polarization responses. By transitioning from single-layer, canonical meta-atoms to bilayer architectures with non-intuitive geometries, our design decouples intrinsic material properties (refractive index and group index), enabling independent engineering of phase and group delays as well as higher-order dispersion properties, while achieving high-efficiency under arbitrary polarization states. We implement a two-stage multi-objective optimization process to generate libraries of meta-atoms, which are then utilized for the rapid design of dispersion-engineered metasurfaces. Additionally, we present a bilayer metasurface stacking technique, paving the way for the realization of high-performance, dispersion-engineered optical devices. Our approach is validated through the demonstration of metasurfaces exhibiting superior chromatic aberration correction and broadband performance, with over 81% averaged efficiency across the 420-nm visible-to-near-infrared bandwidth. Our synergistic combination of advanced design physics, powerful freeform optimization methods, and bi-layer nanofabrication techniques represents a significant breakthrough compared to the state-of-the-art while opening new possibilities for broadband metasurface applications.

physics.app-ph↗

arXiv2024

Refine Large Language Model Fine-tuning via Instruction Vector

Fine-tuning large language models (LLMs) can cause them to lose their general capabilities. However, the intrinsic mechanisms behind such forgetting remain unexplored. In this paper, we begin by examining this phenomenon by focusing on knowledge understanding and instruction following, with the latter identified as the main contributor to forgetting during fine-tuning. Consequently, we propose the Instruction Vector (IV) framework to capture model representations highly related to specific instruction-following capabilities, thereby making it possible to understand model-intrinsic forgetting. Through the analysis of IV dynamics pre and post-training, we suggest that fine-tuning mostly adds specialized reasoning patterns instead of erasing previous skills, which may appear as forgetting. Building on this insight, we develop IV-guided training, which aims to preserve original computation graph, thereby mitigating catastrophic forgetting. Empirical tests on three benchmarks confirm the efficacy of this new approach, supporting the relationship between IVs and forgetting. Our code will be made available soon.

cs.AI↗