arXiv subjects

Tianyi Zhang

Publications and source records attributed to Tianyi Zhang.

At least 127 records · Page 7Linked to original sources

arXiv2026

LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid

Large language models (LLMs) have shown immense potential across various domains, but their high memory requirements and inference costs remain critical challenges for deployment. Post-training quantization (PTQ) has emerged as a promising technique to reduce memory requirements and decoding latency. However, recent accurate quantization methods often depend on specialized computations or custom data formats to achieve better model quality, which limits their compatibility with popular frameworks, as they require dedicated inference kernels tailored to specific hardware and software platforms, hindering wider adoption. Furthermore, many competitive methods have high resource requirements and computational overhead for quantizing models, making it challenging to scale them to hundreds of billions of parameters. In response to these challenges, we propose LeanQuant (Loss-Error-Aware Network Quantization), a novel quantization method that is accurate, versatile, and scalable. In the existing popular iterative loss-error-based quantization framework, we identify a critical limitation in prior methods: the min-max affine quantization grid fails to preserve model quality due to outliers in inverse Hessian diagonals. To overcome this fundamental issue, we propose learning loss-error-aware grids, instead of using non-adaptive min-max affine grids. Our approach not only produces quantized models that are more accurate but also generalizes to a wider range of quantization types, including affine and non-uniform quantization, enhancing compatibility with more frameworks. Extensive experiments with recent LLMs demonstrate that LeanQuant is highly accurate, comparing favorably against competitive baselines in model quality, and scalable, achieving very accurate quantization of Llama-3.1 405B, one of the largest open-source LLMs to date, using two Quadro RTX 8000-48GB GPUs in 21 hours.

cs.LG↗

arXiv2026

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation

Adapting pre-trained large language models (LLMs) is crucial but challenging due to their enormous size. Parameter-efficient fine-tuning (PEFT) techniques typically employ additive adapters applied to frozen model weights. To further reduce memory usage, model weights are often compressed through quantization. However, existing PEFT methods often yield suboptimal model quality because they rely on restrictive assumptions, such as low-rank constraints on adapters to limit the number of trainable parameters. We find that sketching, a popular data compression technique, can serve as an efficient LLM adaptation strategy while avoiding the low-rank assumption. We introduce SketchTune, a compressive adaptation strategy that compresses LLM weights into compact fine-tunable sketches, integrating compression and adaptation into a unified framework. This integration eliminates the need for complex two-path computation in existing PEFT techniques, enabling faster and more memory-efficient training and inference. SketchTune is supported by mathematical insights into matrix classes that are better approximated using sketching rather than low-rank methods. Our extensive evaluations with Llama and Mistral models demonstrate that SketchTune outperforms leading PEFT methods across diverse tasks while using substantially smaller base models and comparable trainable parameters. As a highlight, SketchTune outperforms LoRA, DoRA, and S2FT on commonsense and math benchmarks using 2.6-3.5$\times$ smaller base models and exceeds LoftQ in accuracy by 14.48% on GSM8K with 7.3$\times$ fewer trainable parameters. Our code is available at https://github.com/LeanModels/SketchTune.

cs.LG↗

arXiv2026

70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)

Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs), have grown rapidly in size, creating significant challenges for efficient deployment on resource-constrained hardware. In this paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model. DFloat11 is motivated by the low entropy in the BFloat16 weight representation of LLMs, which reveals significant inefficiency in the existing storage format. By applying entropy coding, DFloat11 assigns dynamic-length encodings to weights based on frequency, achieving near information-optimal compression without any loss of precision. To facilitate efficient inference with dynamic-length encodings, we develop a custom GPU kernel for fast online decompression. Our design incorporates the following: (i) compact, hierarchical lookup tables (LUTs) that fit within GPU SRAM for efficient decoding, (ii) a two-phase GPU kernel for coordinating thread read/write positions using lightweight auxiliary variables, and (iii) transformer-block-level decompression to minimize latency. Experiments on Llama 3.3, Qwen 3, Mistral 3, FLUX.1, and others validate our hypothesis that DFloat11 achieves around 30% model size reduction while preserving bit-for-bit identical outputs. Compared to a potential alternative of offloading parts of an uncompressed model to the CPU to meet memory constraints, DFloat11 achieves 2.3--46.2x higher throughput in token generation. With a fixed GPU memory budget, DFloat11 enables 5.7--14.9x longer generation lengths than uncompressed models. Notably, our method enables lossless inference of Llama 3.1 405B, an 810GB model, on a single node equipped with 8x80GB GPUs.

cs.LG↗

arXiv2025

GraphShaper: Geometry-aware Alignment for Improving Transfer Learning in Text-Attributed Graphs

Graph foundation models represent a transformative paradigm for learning transferable representations across diverse graph domains. Recent methods leverage large language models to unify graph and text modalities into a shared representation space using contrastive learning. However, systematic evaluations reveal significant performance degradation at structural boundaries where distinct topological patterns converge, with accuracy losses exceeding 20 percentage points. This issue arises from a key limitation: current methods assume all graph structures can be encoded within a single Euclidean space. In reality, tree structures require hyperbolic geometry to preserve hierarchical branching, while cyclic patterns depend on spherical geometry for closure properties. At structural boundaries, nodes experience conflicting geometric constraints that uniform encoding spaces cannot resolve. This raises a crucial challenge: \textbf{Can alignment frameworks be designed to respect the intrinsic geometric diversity of graph structures?} We introduce \textbf{GraphShaper}, a geometry-aware framework that enhances graph encoding through multi-geometric specialization. Our approach employs expert networks tailored to different geometric spaces, dynamically computing fusion weights to adaptively integrate geometric properties based on local structural characteristics. This adaptive fusion preserves structural integrity before alignment with text embeddings. Extensive experiments demonstrate that GraphShaper achieves 9.47\% accuracy improvements on citation networks and 7.63\% on social networks in zero-shot settings.

cs.LG↗

arXiv2025

MCI: Multi-Channel Imager on the Chinese Space Station Survey Telescope

The Multi-Channel Imager (MCI) is a powerful near-ultraviolet (NUV) and visible imager onboard the Chinese Space Station Survey Telescope (CSST). The MCI provides three imaging channels, which are the NUV channel, the Blue channel and the Red channel, with the wavelength range of 255-430 nm, 430-700 nm, and 700-1000 nm, respectively. MCI's three channels can target the same field simultaneously, which is unique compared to other imagers onboard the Hubble Space Telescope (HST) or the James Webb Space Telescope (JWST). Each channel employs a CCD focal plane of 9216 x 9232 pixels and $\sim$7\arcmin.5 x 7\arcmin.5 field of view (FOV), which are about $\gtrsim 4$ times greater than the FOVs of HST imagers. The MCI's three channels feature unprecedented sensitivities and field of views complement the NUV and visible capabilities of the CSST for high-precision photometry and weak-signal detection, which would help build a new standard-star system and the deepest UV-Optical exposures for CSST. Rich filter sets of MCI would help explore other sciences such as local emission line mapping, high-z Ly$α$ emitters searching, etc. Here we present key design features, results of current ground tests, and suggested observing strategies of the MCI.

astro-ph.IM↗

arXiv2025

Menta: A Small Language Model for On-Device Mental Health Prediction

Mental health conditions affect hundreds of millions globally, yet early detection remains limited. While large language models (LLMs) have shown promise in mental health applications, their size and computational demands hinder practical deployment. Small language models (SLMs) offer a lightweight alternative, but their use for social media--based mental health prediction remains largely underexplored. In this study, we introduce Menta, the first optimized SLM fine-tuned specifically for multi-task mental health prediction from social media data. Menta is jointly trained across six classification tasks using a LoRA-based framework, a cross-dataset strategy, and a balanced accuracy--oriented loss. Evaluated against nine state-of-the-art SLM baselines, Menta achieves an average improvement of 15.2\% across tasks covering depression, stress, and suicidality compared with the best-performing non--fine-tuned SLMs. It also achieves higher accuracy on depression and stress classification tasks compared to 13B-parameter LLMs, while being approximately 3.25x smaller. Moreover, we demonstrate real-time, on-device deployment of Menta on an iPhone 15 Pro Max, requiring only approximately 3GB RAM. Supported by a comprehensive benchmark against existing SLMs and LLMs, Menta highlights the potential for scalable, privacy-preserving mental health monitoring. Code is available at: https://hong-labs.github.io/menta-project/

cs.AI↗

arXiv2025

Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining

Low-rank optimization has emerged as a promising approach to enabling memory-efficient training of large language models (LLMs). Existing low-rank optimization methods typically project gradients onto a low-rank subspace, reducing the memory cost of storing optimizer states. A key challenge in these methods is selecting suitable subspaces to ensure an effective optimization trajectory. Most existing approaches select the dominant subspace to preserve gradient information, as this intuitively provides the best approximation. However, we find that in practice, the dominant subspace stops changing during pretraining, thereby constraining weight updates to similar subspaces. In this paper, we propose importance sampling for low-rank optimization in LLM pretraining with a provable convergence guarantee, which the dominant subspace approach does not have. Empirically, we demonstrate that our method significantly outperforms previous methods in LLM pretraining tasks.

cs.LG↗

arXiv2025

Approximating Euclidean Shallow-Light Trees

For a weighted graph $G = (V, E, w)$ and a designated source vertex $s \in V$, a spanning tree that simultaneously approximates a shortest-path tree w.r.t. source $s$ and a minimum spanning tree is called a shallow-light tree (SLT). Specifically, an $(α, β)$-SLT of $G$ w.r.t. $s \in V$ is a spanning tree of $G$ with root-stretch $α$ (preserving all distances between $s$ and the other vertices up to a factor of $α$) and lightness $β$ (its weight is at most $β$ times the weight of a minimum spanning tree of $G$). Despite the large body of work on SLTs, the basic question of whether a better approximation algorithm exists was left untouched to date, and this holds in any graph family. This paper makes a first nontrivial step towards this question by presenting two bicriteria approximation algorithms. For any $ε>0$, a set $P$ of $n$ points in constant-dimensional Euclidean space and a source $s\in P$, our first (respectively, second) algorithm returns, in $O(n \log n \cdot {\rm polylog}(1/ε))$ time, a non-Steiner (resp., Steiner) tree with root-stretch $1+O(ε\log ε^{-1})$ and weight at most $O(\mathrm{opt}_ε\cdot \log^2 ε^{-1})$ (resp., $O(\mathrm{opt}_ε\cdot \log ε^{-1})$), where $\mathrm{opt}_ε$ denotes the minimum weight of a non-Steiner (resp., Steiner) tree with root-stretch $1+ε$.

cs.CG↗

arXiv2025

Behavioral Indicators of Loneliness: Predicting University Students' Loneliness Scores from Smartphone Sensing Data

Loneliness is a critical mental health issue among university students, yet traditional monitoring methods rely primarily on retrospective self-reports and often lack real-time behavioral context. This study explores the use of passive smartphone sensing data to predict loneliness levels, addressing the limitations of existing approaches in capturing its dynamic nature. We integrate smartphone sensing with machine learning and large language models respectively to develop generalized and personalized models. Our Random Forest generalized models achieved mean absolute errors of 3.29 at midterm and 3.98 (out of 32) at the end of semester on the UCLA Loneliness Scale (short form), identifying smartphone screen usage and location mobility to be key predictors. The one-shot approach leveraging large language models reduced prediction errors by up to 42% compared to zero-shot inference. The one-shot results from personalized models highlighted screen usage, application usage, battery, and location transitions as salient behavioral indicators. These findings demonstrate the potential of smartphone sensing data for scalable and interpretable loneliness detection in digital mental health.

cs.HC↗

arXiv2025

LLM-Driven Kernel Evolution: Automating Driver Updates in Linux

Linux kernel evolution breaks drivers through API/ABI changes, semantic shifts, and security-hardening updates. We introduce DRIVEBENCH, an executable corpus of kernel$\rightarrow$driver co-evolution cases, and AUTODRIVER, a closed-loop, LLM-driven system for automating driver maintenance. The system integrates prompt engineering, multi-agent collaboration, static analysis, and iterative validation to ensure that generated patches are not only syntactically correct but also functionally and semantically consistent with kernel conventions. The corpus spans v5.10-v6.10 with 235 validated cases drawn from 612 candidates. In evaluation across 55 cases, AUTODRIVER achieves 56.4% compilation success; QEMU-based boot verification indicates that compiled patches preserve driver initialization in most instances. By releasing DRIVEBENCH and tooling, we enable reproducible research and a practical route to continuous, safe co-evolution of drivers with the Linux kernel.

cs.SE↗

arXiv2025

Visionary Co-Driver: Enhancing Driver Perception of Potential Risks with LLM and HUD

Drivers' perception of risky situations has always been a challenge in driving. Existing risk-detection methods excel at identifying collisions but face challenges in assessing the behavior of road users in non-collision situations. This paper introduces Visionary Co-Driver, a system that leverages large language models to identify non-collision roadside risks and alert drivers based on their eye movements. Specifically, the system combines video processing algorithms and LLMs to identify potentially risky road users. These risks are dynamically indicated on an adaptive heads-up display interface to enhance drivers' attention. A user study with 41 drivers confirms that Visionary Co-Driver improves drivers' risk perception and supports their recognition of roadside risks.

cs.HC↗

arXiv2025

Text-to-SQL Domain Adaptation via Human-LLM Collaborative Data Annotation

Text-to-SQL models, which parse natural language (NL) questions to executable SQL queries, are increasingly adopted in real-world applications. However, deploying such models in the real world often requires adapting them to the highly specialized database schemas used in specific applications. We find that existing text-to-SQL models experience significant performance drops when applied to new schemas, primarily due to the lack of domain-specific data for fine-tuning. This data scarcity also limits the ability to effectively evaluate model performance in new domains. Continuously obtaining high-quality text-to-SQL data for evolving schemas is prohibitively expensive in real-world scenarios. To bridge this gap, we propose SQLsynth, a human-in-the-loop text-to-SQL data annotation system. SQLsynth streamlines the creation of high-quality text-to-SQL datasets through human-LLM collaboration in a structured workflow. A within-subjects user study comparing SQLsynth with manual annotation and ChatGPT shows that SQLsynth significantly accelerates text-to-SQL data annotation, reduces cognitive load, and produces datasets that are more accurate, natural, and diverse. Our code is available at https://github.com/magic-YuanTian/SQLsynth.

cs.HC↗

arXiv2025

Personalized Chain-of-Thought Summarization of Financial News for Investor Decision Support

Financial advisors and investors struggle with information overload from financial news, where irrelevant content and noise obscure key market signals and hinder timely investment decisions. To address this, we propose a novel Chain-of-Thought (CoT) summarization framework that condenses financial news into concise, event-driven summaries. The framework integrates user-specified keywords to generate personalized outputs, ensuring that only the most relevant contexts are highlighted. These personalized summaries provide an intermediate layer that supports language models in producing investor-focused narratives, bridging the gap between raw news and actionable insights.

q-fin.GN↗

arXiv2025

Individualized Cognitive Simulation in Large Language Models: Evaluating Different Cognitive Representation Methods

Individualized cognitive simulation (ICS) aims to build computational models that approximate the thought processes of specific individuals. While large language models (LLMs) convincingly mimic surface-level human behavior such as role-play, their ability to simulate deeper individualized cognitive processes remains poorly understood. To address this gap, we introduce a novel task that evaluates different cognitive representation methods in ICS. We construct a dataset from recently published novels (later than the release date of the tested LLMs) and propose an 11-condition cognitive evaluation framework to benchmark seven off-the-shelf LLMs in the context of authorial style emulation. We hypothesize that effective cognitive representations can help LLMs generate storytelling that better mirrors the original author. Thus, we test different cognitive representations, e.g., linguistic features, concept mappings, and profile-based information. Results show that combining conceptual and linguistic features is particularly effective in ICS, outperforming static profile-based cues in overall evaluation. Importantly, LLMs are more effective at mimicking linguistic style than narrative structure, underscoring their limits in deeper cognitive simulation. These findings provide a foundation for developing AI systems that adapt to individual ways of thinking and expression, advancing more personalized and human-aligned creative technologies.

cs.AI↗

arXiv2025

Predictive Indicator of Critical Point in Equilibrium and Nonequilibrium Magnetic Systems

Determining critical points of phase transitions from partial data is essential to avoid abrupt system collapses and reducing experimental or computational costs. However, the complex physical systems and phase transition phenomena have long hindered the development of unified approaches applicable to both equilibrium and nonequilibrium phase transitions. In this work, we propose predictive indicators to determine critical points in equilibrium and nonequilibrium magnetic systems based on frequency-dependent response function. For equilibrium phase transition, such as magnetization switching under magnetic field, the static magnetization response function to a perturbing magnetic field diverges at the critical field, serving as a noise-resilient predictive indicator that also reflects the transition order and critical exponents. In contrast, for nonequilibrium phase transition, such as magnetization switching driven by spin-transfer torque, static response fails to signal criticality. Instead, the dynamic response at ferromagnetic resonance frequency diverges at the critical point, which is also robust against thermal noise. We further demonstrate that these static and dynamic indicators can be unified in the frame of first-order linear differential systems, offering a generalizable strategy for predicting criticality in both equilibrium and nonequilibrium transitions.

cond-mat.stat-mech↗

arXiv2025

IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation

Continual pretraining promises to adapt large language models (LLMs) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to instruction-tuned models is known to degrade their instruction-following capability and semantic representations. Existing fixes assume access to the original base model or rely on knowledge from an external domain-specific database - both of which pose a realistic barrier in settings where the base model weights are withheld for safety reasons or reliable external corpora are unavailable. In this work, we propose Instruction-Knowledge-Aware Continual Adaptation (IKnow), a simple and general framework that formulates novel self-supervised objectives in the instruction-response dialogue format. Rather than depend- ing on external resources, IKnow leverages domain knowledge embedded within the text itself and learns to encode it at a deeper semantic level.

cs.AI↗

arXiv2025

Approximate Light Spanners in Planar Graphs

In their seminal paper, Althöfer et al. (DCG 1993) introduced the {\em greedy spanner} and showed that, for any weighted planar graph $G$, the weight of the greedy $(1+ε)$-spanner is at most $(1+\frac{2}ε) \cdot w(MST(G))$, where $w(MST(G))$ is the weight of a minimum spanning tree $MST(G)$ of $G$. This bound is optimal in an {\em existential sense}: there exist planar graphs $G$ for which any $(1+ε)$-spanner has a weight of at least $(1+\frac{2}ε) \cdot w(MST(G))$. However, as an {\em approximation algorithm}, even for a {\em bicriteria} approximation, the weight approximation factor of the greedy spanner is essentially as large as the existential bound: There exist planar graphs $G$ for which the greedy $(1+x ε)$-spanner (for any $1\leq x = O(ε^{-1/2})$) has a weight of $Ω(\frac{1}{ε\cdot x^2})\cdot w(G_{OPT, ε})$, where $G_{OPT, ε}$ is a $(1+ε)$-spanner of $G$ of minimum weight. Despite the flurry of works over the past three decades on approximation algorithms for spanners as well as on light(-weight) spanners, there is still no (possibly bicriteria) approximation algorithm for light spanners in weighted planar graphs that outperforms the existential bound. As our main contribution, we present a polynomial time algorithm for constructing, in any weighted planar graph $G$, a $(1+ε\cdot 2^{O(\log^* 1/ε)})$-spanner for $G$ of total weight $O(1)\cdot w(G_{OPT, ε})$. To achieve this result, we develop a new technique, which we refer to as {\em iterative planar pruning}. It iteratively modifies a spanner [...]

cs.DS↗

arXiv2025

Vizing's Theorem in Deterministic Almost-Linear Time

Vizing's theorem states that any $n$-vertex $m$-edge graph of maximum degree $Δ$ can be edge colored using at most $Δ+ 1$ different colors. Vizing's original proof is easily translated into a deterministic $O(mn)$ time algorithm. This deterministic time bound was subsequently improved to $\tilde O(m \sqrt n)$ time, independently by [Arjomandi, 1982] and by [Gabow et al., 1985]. A series of recent papers improved the time bound of $\tilde O(m\sqrt{n})$ using randomization, culminating in the randomized near-linear time $(Δ+1)$-coloring algorithm by [Assadi, Behnezhad, Bhattacharya, Costa, Solomon, and Zhang, 2025]. At the heart of all of these recent improvements, there is some form of a sublinear time algorithm. Unfortunately, sublinear time algorithms as a whole almost always require randomization. This raises a natural question: can the deterministic time complexity of the problem be reduced below the $\tilde O(m\sqrt{n})$ barrier? In this paper, we answer this question in the affirmative. We present a deterministic almost-linear time $(Δ+1)$-coloring algorithm, namely, an algorithm running in $m \cdot 2^{O(\sqrt{\log Δ})} \cdot \log n = m^{1+o(1)}$ time. Our main technical contribution is to entirely forego sublinear time algorithms. We do so by presenting a new deterministic color-type sparsification approach that runs in almost-linear (instead of sublinear) time, but can be used to color a much larger set of edges.

cs.DS↗