arXiv subjects

Xiangyu Zhang

Publications and source records attributed to Xiangyu Zhang.

At least 181 records · Page 10Linked to original sources

arXiv2024

Reconstructive Visual Instruction Tuning

This paper introduces reconstructive visual instruction tuning (ROSS), a family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals. In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, ROSS prompts LMMs to supervise visual outputs via reconstructing input images. By doing so, it capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals. To address this issue, ROSS employs a denoising objective to reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values. This intrinsic activation design inherently encourages LMMs to maintain image detail, thereby enhancing their fine-grained comprehension capabilities and reducing hallucinations. Empirically, ROSS consistently brings significant improvements across different visual encoders and language models. In comparison with extrinsic assistance state-of-the-art alternatives that aggregate multiple visual experts, ROSS delivers competitive performance with a single SigLIP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs.

cs.CV↗

arXiv2024

Controllable Thermo-Stimulated Luminescence in Niobate Persistent Phosphor by Constructing the Photovoltaic/Electrolytic Cell for Remote Intelligent Anti-Counterfeiting

Persistent luminescence (PersL) carrying remote key information plays a crucial role for intelligent anti-counterfeiting applications. However, the weak PersL intensity accompanied by uncontrollability limits their practical application. Here we develop LiNbO3 (LNO):Pr,Bi phosphor with enhanced red PersL by trace doping Sm3+. The LNO:Pr,Bi,Sm phosphor exhibits quadruplet luminescence, including polychrome photoluminescence, PersL, and photo/thermo-stimulated luminescence (PSL/TSL). Particularly, the enhanced TSL can carry remote subjective information independent of the phosphor itself by controlling the temperature. A mechanism of afterglow enhancement is proposed based on constructing reversible photovoltaic cells and electrolytic cells by photothermal redox reactions using Bi3+ + VO and Bi3+/Pr3+ + VLi' ion pair. This study has sparked the exploration of designing the information storage PersL materials for more sophisticated remote intelligent anti-counterfeiting.

physics.optics↗

arXiv2024

SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Control

Autonomous driving progress relies on large-scale annotated datasets. In this work, we explore the potential of generative models to produce vast quantities of freely-labeled data for autonomous driving applications and present SubjectDrive, the first model proven to scale generative data production in a way that could continuously improve autonomous driving applications. We investigate the impact of scaling up the quantity of generative data on the performance of downstream perception models and find that enhancing data diversity plays a crucial role in effectively scaling generative data production. Therefore, we have developed a novel model equipped with a subject control mechanism, which allows the generative model to leverage diverse external data sources for producing varied and useful data. Extensive evaluations confirm SubjectDrive's efficacy in generating scalable autonomous driving training data, marking a significant step toward revolutionizing data production methods in this field.

cs.CV↗

arXiv2024

Enhancement of piezoelectric response in V doped LiNbO3 films deposited by RF magnetron sputtering

LiNbO3 films doped with vanadium (V) were deposited using RF magnetron sputtering technique. To realize doping with a wider range of V concentration, a 30 mm V metal inlaid target asymmetrically embedded in the 150 mm lithium niobate target was used. The V concentration in the deposited films was a decreasing function of the distance from the V target. The V/Nb ratio in the film decreased from 0.155 to 0.024. Surface and inner morphology and structure, phase and element composition, microstructure, and ferroelectric properties of the undoped and V doped LiNbO3 films were studied. The measured maximal d33 constant of the LiNbVO film with V/Nb ratio of 0.07 was about three times higher than that of the undoped LiNbO3 film, 13.5 pC/N and 4.76 pC/N, respectively. The optimal composition in the deposition geometry used was within the V/Nb ratio range of 0.05 to 0.13. Undoped and V doped LiNbO3 thin films were used as bulk acoustic wave ultrasonic transducers deposited on stainless steel plates to generate longitudinal waves and compare their ultrasonic performance.

cond-mat.mtrl-sci↗

arXiv2024

Align-DETR: Enhancing End-to-end Object Detection with Aligned Loss

DETR has set up a simple end-to-end pipeline for object detection by formulating this task as a set prediction problem, showing promising potential. Despite its notable advancements, this paper identifies two key forms of misalignment within the model: classification-regression misalignment and cross-layer target misalignment. Both issues impede DETR's convergence and degrade its overall performance. To tackle both issues simultaneously, we introduce a novel loss function, termed as Align Loss, designed to resolve the discrepancy between the two tasks. Align Loss guides the optimization of DETR through a joint quality metric, strengthening the connection between classification and regression. Furthermore, it incorporates an exponential down-weighting term to facilitate a smooth transition from positive to negative samples. Align-DETR also employs many-to-one matching for supervision of intermediate layers, akin to the design of H-DETR, which enhances robustness against instability. We conducted extensive experiments, yielding highly competitive results. Notably, our method achieves a 49.3% (+0.6) AP on the H-DETR baseline with the ResNet-50 backbone. It also sets a new state-of-the-art performance, reaching 50.5% AP in the 1x setting and 51.7% AP in the 2x setting, surpassing several strong competitors. Our code is available at https://github.com/FelixCaae/AlignDETR.

cs.CV↗

arXiv2024

Fixing Large Language Models' Specification Misunderstanding for Better Code Generation

Code generation is to automatically generate source code conforming to a given programming specification, which has received extensive attention especially with the development of large language models (LLMs). Due to the inherent difficulty of code generation, the code generated by LLMs may not be aligned with the specification. Although thought-eliciting prompting techniques have been proposed to enhance the code generation performance of LLMs, producing correct understanding for complicated programming problems remains challenging, resulting in unsatisfactory performance. Also, some feedback-based prompting techniques have been proposed to fix incorrect code using error messages produced by test execution. However, when the generated code deviates significantly from the ground truth, they encounter difficulties in improving performance based on such coarse-grained information. In this work, we propose a novel prompting technique, called μFiX, to improve the code generation performance of LLMs by devising both sophisticated thought-eliciting prompting and feedback-based prompting and making the first exploration on their synergy. It first exploits test case analysis to obtain specification understanding and enables a self-improvement process to identify and refine the misunderstanding in the thought-eliciting prompting phase. μFiX further fixes the specification understanding towards the direction reducing the gap between the provided understanding (from the first phase) and the actual understanding implicitly utilized by LLMs for code generation in the feedback-based prompting phase. By improving the understanding with μFiX, the code generation performance of LLMs can be largely improved. Our evaluation on two advanced LLMs (ChatGPT and DeepSeek-Coder) with six widely-used benchmarks by comparing with 15 baselines, demonstrates the effectiveness of μFiX ...

cs.SE↗

arXiv2024

Symbol Preference Aware Generative Models for Recovering Variable Names from Stripped Binary

Decompilation aims to recover the source code form of a binary executable. It has many security applications, such as malware analysis, vulnerability detection, and code hardening. A prominent challenge in decompilation is to recover variable names. We propose a novel technique that leverages the strengths of generative models while mitigating model biases. We build a prototype, GenNm, from pre-trained generative models CodeGemma-2B, CodeLlama-7B, and CodeLlama-34B. We finetune GenNm on decompiled functions and teach models to leverage contextual information. GenNm includes names from callers and callees while querying a function, providing rich contextual information within the model's input token limitation. We mitigate model biases by aligning the output distribution of models with symbol preferences of developers. Our results show that GenNm improves the state-of-the-art name recovery precision by 5.6-11.4 percentage points on two commonly used datasets and improves the state-of-the-art by 32% (from 17.3% to 22.8%) in the most challenging setup where ground-truth variable names are not seen in the training dataset.

cs.SE↗

arXiv2024

LLMDFA: Analyzing Dataflow in Code with Large Language Models

Dataflow analysis is a fundamental code analysis technique that identifies dependencies between program values. Traditional approaches typically necessitate successful compilation and expert customization, hindering their applicability and usability for analyzing uncompilable programs with evolving analysis needs in real-world scenarios. This paper presents LLMDFA, an LLM-powered compilation-free and customizable dataflow analysis framework. To address hallucinations for reliable results, we decompose the problem into several subtasks and introduce a series of novel strategies. Specifically, we leverage LLMs to synthesize code that outsources delicate reasoning to external expert tools, such as using a parsing library to extract program values of interest and invoking an automated theorem prover to validate path feasibility. Additionally, we adopt a few-shot chain-of-thought prompting to summarize dataflow facts in individual functions, aligning the LLMs with the program semantics of small code snippets to mitigate hallucinations. We evaluate LLMDFA on synthetic programs to detect three representative types of bugs and on real-world Android applications for customized bug detection. On average, LLMDFA achieves 87.10% precision and 80.77% recall, surpassing existing techniques with F1 score improvements of up to 0.35. We have open-sourced LLMDFA at https://github.com/chengpeng-wang/LLMDFA.

cs.PL↗

arXiv2024

Enhancing Molecular Design through Graph-based Topological Reinforcement Learning

The generation of drug-like molecules is crucial for drug design. Existing reinforcement learning (RL) methods often overlook structural information. However, feature engineering-based methods usually merely focus on binding affinity prediction without substantial molecular modification. To address this, we present Graph-based Topological Reinforcement Learning (GraphTRL), which integrates both chemical and structural data for improved molecular generation. GraphTRL leverages multiscale weighted colored graphs (MWCG) and persistent homology, combined with molecular fingerprints, as the state space for RL. Evaluations show that GraphTRL outperforms existing methods in binding affinity prediction, offering a promising approach to accelerate drug discovery.

cs.LG↗

arXiv2024

Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models

Instruction tuning constitutes a prevalent technique for tailoring Large Vision Language Models (LVLMs) to meet individual task requirements. To date, most of the existing approaches are confined to single-task adaptation, whereas the requirements in real-world scenarios are inherently varied and continually evolving. Thus an ideal LVLM should sustain continual instruction tuning in the face of stream-task distributions (i.e., different domains, emerging capabilities, and new datasets) while minimizing the forgetting of previously acquired knowledge. To achieve this, we propose a new benchmark for COntinuAl inStruction Tuning on LVLMs (COAST), which encompasses the aforementioned domain-incremental, capability-incremental, and dataset-incremental configurations. In terms of methodology, we propose Continual LLaVA, a rehearsal-free method tailored for continual instruction tuning in LVLMs. To circumvent the additional overhead associated with experience replay, we freeze LVLMs and construct the dual increment embeddings for each input instruction to facilitate parameter-efficient tuning. Specifically, the increment embeddings can be decomposed into two principal components: 1) intrinsic increment embeddings to encode task-specific characteristics. To achieve this, we set up a low-rank pool containing candidate embeddings, from which we select the relevant ones based on their similarity with the user instructions; 2) contextual increment embeddings to investigate the inter-dependencies across tasks. In this regard, the low-rank embeddings chosen in the previous tasks are aggregated via learnable weighted sum to provide complementary hints. Extensive experiments indicate that the proposed Continual LLaVA outperforms previous methods by significantly reducing the forgetting during the continual instruction tuning process.

cs.CV↗

arXiv2024

Anchor Attention, Small Cache: Code Generation with Large Language Models

The development of large language models (LLMs) has revolutionized automated code generation. However, their high demand of computation resources has hindered a broader deployment and raised environmental concerns. A common strategy for diminishing computational demands is to cache Key-Value (KV) states from the attention mechanism which is adopted predominately by mainstream LLMs. It can mitigate the need of repeated attention computations, but brings significant memory overhead. Current practices in NLP often use sparse attention which may, unfortunately, lead to substantial inaccuracies, or hallucinations, in code generation tasks. In this paper, we analyze the attention weights distribution within code generation models via an empirical study, uncovering a sparsity pattern, i.e., the aggregation of information at specific anchor points. Based on this observation, we propose a novel approach, AnchorCoder, which features token-wise anchor attention designed to extract and compress the contextual information, and layer-wise anchor attention enabling cross-layer communication to mitigate the issue of excessive superposition caused by the compression. The extensive experiments across multiple benchmark datasets confirm the effectiveness of AnchorCoder, which can consistently achieve a significant (at least 70%) reduction in KV cache requirements, while preserving the majority of model's performance.

cs.SE↗

arXiv2024

Selective State Space Model for Monaural Speech Enhancement

Voice user interfaces (VUIs) have facilitated the efficient interactions between humans and machines through spoken commands. Since real-word acoustic scenes are complex, speech enhancement plays a critical role for robust VUI. Transformer and its variants, such as Conformer, have demonstrated cutting-edge results in speech enhancement. However, both of them suffers from the quadratic computational complexity with respect to the sequence length, which hampers their ability to handle long sequences. Recently a novel State Space Model called Mamba, which shows strong capability to handle long sequences with linear complexity, offers a solution to address this challenge. In this paper, we propose a novel hybrid convolution-Mamba backbone, denoted as MambaDC, for speech enhancement. Our MambaDC marries the benefits of convolutional networks to model the local interactions and Mamba's ability for modeling long-range global dependencies. We conduct comprehensive experiments within both basic and state-of-the-art (SoTA) speech enhancement frameworks, on two commonly used training targets. The results demonstrate that MambaDC outperforms Transformer, Conformer, and the standard Mamba across all training targets. Built upon the current advanced framework, the use of MambaDC backbone showcases superior results compared to existing \textcolor{black}{SoTA} systems. This sets the stage for efficient long-range global modeling in speech enhancement.

eess.AS↗

arXiv2024

Source Code Foundation Models are Transferable Binary Analysis Knowledge Bases

Human-Oriented Binary Reverse Engineering (HOBRE) lies at the intersection of binary and source code, aiming to lift binary code to human-readable content relevant to source code, thereby bridging the binary-source semantic gap. Recent advancements in uni-modal code model pre-training, particularly in generative Source Code Foundation Models (SCFMs) and binary understanding models, have laid the groundwork for transfer learning applicable to HOBRE. However, existing approaches for HOBRE rely heavily on uni-modal models like SCFMs for supervised fine-tuning or general LLMs for prompting, resulting in sub-optimal performance. Inspired by recent progress in large multi-modal models, we propose that it is possible to harness the strengths of uni-modal code models from both sides to bridge the semantic gap effectively. In this paper, we introduce a novel probe-and-recover framework that incorporates a binary-source encoder-decoder model and black-box LLMs for binary analysis. Our approach leverages the pre-trained knowledge within SCFMs to synthesize relevant, symbol-rich code fragments as context. This additional context enables black-box LLMs to enhance recovery accuracy. We demonstrate significant improvements in zero-shot binary summarization and binary function name recovery, with a 10.3% relative gain in CHRF and a 16.7% relative gain in a GPT4-based metric for summarization, as well as a 6.7% and 7.4% absolute increase in token-level precision and recall for name recovery, respectively. These results highlight the effectiveness of our approach in automating and improving binary code analysis.

cs.SE↗

arXiv2024

Dust extinction-curve variation in the translucent interstellar medium is driven by PAH growth

The first all-sky, high-resolution, 3D map of the optical extinction curve of the Milky Way (Zhang & Green 2024) revealed an unexpected steepening of the extinction curve in the moderate-density, "translucent" interstellar medium (ISM). We argue that this trend is driven by growth of polycyclic aromatic hydrocarbons (PAHs) through gas-phase accretion. We find a strong anti-correlation between the slope of the optical extinction curve -- parameterized by $R(V)$ -- and maps of PAH abundance -- parameterized by $q_{\rm PAH}$ -- derived from infrared emission. The range of observed $q_{\rm PAH}$ indicates PAH growth by a factor of $\sim$2 between $A_V \simeq 1$ and 3. This implies a factor-of-two stronger 2175 Angstrom feature, which is sufficient to lower $R(V)$ by the observed amount. This level of PAH growth is possible given rapid accretion timescales and the depletion of carbon in the translucent ISM. Spectral observations by JWST would provide a definitive test of this proposed explanation of $R(V)$ variation.

astro-ph.GA↗

arXiv2024

High-Throughput Information Storage in An Intelligent Response Phosphor

Persistent phosphor has emerged as a promising candidate for information storage due to the rapid accessibility and low-energy requirements. However, the low storage capacity has limited its practical application. Herein, we skillfully designed and developed NaGdGeO4:Pb2+,Tb3+ stimulated phosphor by trace doped Sm3+. As expected, this phosphor demonstrates the larger carrier capacity than traditional commercial SrAl2O4:Eu2+,Dy3+ phosphors and super-strong thermo-stimulated luminescence (TSL) that is three times greater than its photoluminescence (PL) intensity (PL efficiency: 17.3%). A mechanism of the enhanced and controllable TSL is proposed based on electron-hole defect pair structure. We further present a high-throughput optical data recording in five dimensions in a single fluorescent film recording layer. The findings described here provides not only a universal approach for construction TSL materials, but also a new paradigm for future generation optical storage technology.

physics.optics↗

arXiv2024

m-weak group MP inverse

In this paper, we introduce a new matrix decomposition called the m-Core-nilpotent decomposition which is an extension of the Core-nilpotent decomposition. By this new decomposition, we propose a new generalized inverse named the m-weak group MP inverse which unifies the DMP-inverse and weak core inverse. Some characterizations, properties and representations of the m-weak group MP inverse are presented. In addition, the proposed generalized inverse is applicable to solving a restricted matrix equation.

math.RA↗

arXiv2024

Quantum Phonon Dynamics Induced Spontaneous Spin-Orbit Coupling

Spin-orbit coupling in solids is typically a single-body effect arising from relativity. In this work, we propose a spontaneous generation of spin-orbit coupling from symmetry breaking. A spin-dependent electron-phonon coupling model is investigated on a half-filled square lattice, which is solved by sign-problem-free quantum Monte Carlo simulations. The phase diagram as function of phonon frequency $ω$ and coupling constant $λ$ is fully investigated. The spin-orbit coupling emerges as an order in the ground state for any $λ$ in the adiabatic limit, accompanied by a breathing mode of lattice distortion and a staggered loop spin-current. This phase dominates in the entire range of $ω$ with $λ< λ_{\infty}$, a critical value in the $ω\to \infty$ limit. With increasing $ω$ and $λ> λ_{\infty}$, the emergent spin-orbit coupling is suppressed and a phase transition occurs leading to charge-density-wave degenerate with superconductivity order. Our work opens up the possibility of hidden spin-orbit coupling in materials where it is otherwise forbidden by lattice symmetry and paves the way to explore new usable materials or devices in spintronics.

cond-mat.str-el↗

arXiv2024

ASPIRER: Bypassing System Prompts With Permutation-based Backdoors in LLMs

Large Language Models (LLMs) have become integral to many applications, with system prompts serving as a key mechanism to regulate model behavior and ensure ethical outputs. In this paper, we introduce a novel backdoor attack that systematically bypasses these system prompts, posing significant risks to the AI supply chain. Under normal conditions, the model adheres strictly to its system prompts. However, our backdoor allows malicious actors to circumvent these safeguards when triggered. Specifically, we explore a scenario where an LLM provider embeds a covert trigger within the base model. A downstream deployer, unaware of the hidden trigger, fine-tunes the model and offers it as a service to users. Malicious actors can purchase the trigger from the provider and use it to exploit the deployed model, disabling system prompts and achieving restricted outcomes. Our attack utilizes a permutation trigger, which activates only when its components are arranged in a precise order, making it computationally challenging to detect or reverse-engineer. We evaluate our approach on five state-of-the-art models, demonstrating that our method achieves an attack success rate (ASR) of up to 99.50% while maintaining a clean accuracy (CACC) of 98.58%, even after defensive fine-tuning. These findings highlight critical vulnerabilities in LLM deployment pipelines and underscore the need for stronger defenses.

cs.CR↗