arXiv subjects

Zhiqiang Shen

Publications and source records attributed to Zhiqiang Shen.

At least 91 records · Page 5Linked to original sources

arXiv2025

ConStyX: Content Style Augmentation for Generalizable Medical Image Segmentation

Medical images are usually collected from multiple domains, leading to domain shifts that impair the performance of medical image segmentation models. Domain Generalization (DG) aims to address this issue by training a robust model with strong generalizability. Recently, numerous domain randomization-based DG methods have been proposed. However, these methods suffer from the following limitations: 1) constrained efficiency of domain randomization due to their exclusive dependence on image style perturbation, and 2) neglect of the adverse effects of over-augmented images on model training. To address these issues, we propose a novel domain randomization-based DG method, called content style augmentation (ConStyX), for generalizable medical image segmentation. Specifically, ConStyX 1) augments the content and style of training data, allowing the augmented training data to better cover a wider range of data domains, and 2) leverages well-augmented features while mitigating the negative effects of over-augmented features during model training. Extensive experiments across multiple domains demonstrate that our ConStyX achieves superior generalization performance. The code is available at https://github.com/jwxsp1/ConStyX.

eess.IV↗

arXiv2025

Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering

Large Language Models (LLMs) have shown strong inductive reasoning ability across various domains, but their reliability is hindered by the outdated knowledge and hallucinations. Retrieval-Augmented Generation mitigates these issues by grounding LLMs with external knowledge; however, most existing RAG pipelines rely on unstructured text, limiting interpretability and structured reasoning. Knowledge graphs, which represent facts as relational triples, offer a more structured and compact alternative. Recent studies have explored integrating knowledge graphs with LLMs for knowledge graph question answering (KGQA), with a significant proportion adopting the retrieve-then-reasoning paradigm. In this framework, graph-based retrievers have demonstrated strong empirical performance, yet they still face challenges in generalization ability. In this work, we propose RAPL, a novel framework for efficient and effective graph retrieval in KGQA. RAPL addresses these limitations through three aspects: (1) a two-stage labeling strategy that combines heuristic signals with parametric models to provide causally grounded supervision; (2) a model-agnostic graph transformation approach to capture both intra- and inter-triple interactions, thereby enhancing representational capacity; and (3) a path-based reasoning strategy that facilitates learning from the injected rational knowledge, and supports downstream reasoner through structured inputs. Empirically, RAPL outperforms state-of-the-art methods by $2.66\%-20.34\%$, and significantly reduces the performance gap between smaller and more powerful LLM-based reasoners, as well as the gap under cross-dataset settings, highlighting its superior retrieval capability and generalizability. Codes are available at: https://github.com/tianyao-aka/RAPL.

cs.CL↗

arXiv2025

DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation

Recent advances in dataset distillation have led to solutions in two main directions. The conventional batch-to-batch matching mechanism is ideal for small-scale datasets and includes bi-level optimization methods on models and syntheses, such as FRePo, RCIG, and RaT-BPTT, as well as other methods like distribution matching, gradient matching, and weight trajectory matching. Conversely, batch-to-global matching typifies decoupled methods, which are particularly advantageous for large-scale datasets. This approach has garnered substantial interest within the community, as seen in SRe$^2$L, G-VBSM, WMDD, and CDA. A primary challenge with the second approach is the lack of diversity among syntheses within each class since samples are optimized independently and the same global supervision signals are reused across different synthetic images. In this study, we propose a new Diversity-driven EarlyLate Training (DELT) scheme to enhance the diversity of images in batch-to-global matching with less computation. Our approach is conceptually simple yet effective, it partitions predefined IPC samples into smaller subtasks and employs local optimizations to distill each subset into distributions from distinct phases, reducing the uniformity induced by the unified optimization process. These distilled images from the subtasks demonstrate effective generalization when applied to the entire task. We conduct extensive experiments on CIFAR, Tiny-ImageNet, ImageNet-1K, and its sub-datasets. Our approach outperforms the previous state-of-the-art by 2$\sim$5% on average across different datasets and IPCs (images per class), increasing diversity per class by more than 5% while reducing synthesis time by up to 39.3% for enhancing the training efficiency. Code is available at: https://github.com/VILA-Lab/DELT.

cs.CV↗

arXiv2025

Network of velocity-coherent filaments formed by supersonic turbulence in a very-high-velocity HI cloud

The warm neutral medium (WNM) was thought to be subsonically/transonically turbulent, and it lacks a network of intertwined filaments that are commonly seen in both molecular clouds and cold neutral medium (CNM). Here, we report HI~21 cm line observations of a very-high-velocity (-330 km s$^{-1}$ $<V_{\rm LSR}<$ -250 km s$^{-1}$) cloud (VHVC), using the Five-hundred-meter Aperture Spherical radio Telescope (FAST) with unprecedented resolution and sensitivity. For the first time, such a VHVC is clearly revealed to be a supersonic WNM system consisting of a network of velocity-coherent HI~filaments. The filaments are in the forms of slim curves, hubs, and webs, distributed in different layers within the position-position-velocity ({\it ppv}) data cube. The entire cloud has skewed log-normal probability distribution of column density and the filaments themselves show asymmetrical radial density profiles, indicating shock compression by supersonic magnetohydrodynamic (MHD) turbulence, as is also confirmed by our MHD simulation (sonic Mach number $M_{\rm s}=3$ and Alfvén Mach number $M_{\rm A}=1$). This work suggests that hierarchical filaments can be established by shocks in a low-density WNM, where gravity is negligible, offering a viable pathway to structure formation in the earliest evolutionary phases of the interstellar medium (ISM).

astro-ph.GA↗

arXiv2025

The Chinese Pulsar Timing Array data release I. Single pulsar noise analysis

The Chinese Pulsar Timing Array (CPTA) has collected observations from 57 millisecond pulsars using the Five-hundred-meter Aperture Spherical Radio Telescope (FAST) for close to three years, for the purpose of searching for gravitational waves (GWs). To robustly search for ultra-low-frequency GWs, pulsar timing arrays (PTAs) need to use models to describe the noise from the individual pulsars. We report on the results from the single pulsar noise analysis of the CPTA data release I (DR1). Conventionally, power laws in the frequency domain are used to describe pulsar red noise and dispersion measurement (DM) variations over time. Employing Bayesian methods, we found the choice of number and range of frequency bins with the highest evidence for each pulsar individually. A comparison between a dataset using DM piecewise measured (DMX) values and a power-law Gaussian process to describe the DM variations shows strong Bayesian evidence in favour of the power-law model. Furthermore, we demonstrate that the constraints obtained from four independent software packages are very consistent with each other. The short time span of the CPTA DR1, paired with the large sensitivity of FAST, has proved to be a challenge for the conventional noise model using a power law. This mainly shows in the difficulty to separate different noise terms due to their covariances with each other. Nineteen pulsars are found to display covariances between the short-term white noise and long-term red and DM noise. With future CPTA datasets, we expect that the degeneracy can be broken. Finally, we compared the CPTA DR1 results against the noise properties found by other PTA collaborations. While we can see broad agreement, there is some tension between different PTA datasets for some of the overlapping pulsars. This could be due to the differences in the methods and frequency range compared to the other PTAs.

astro-ph.HE↗

arXiv2025

DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation

Retrieval-Augmented Generation (RAG) methods have proven highly effective for tasks requiring factual consistency and robust knowledge retrieval. However, large-scale RAG systems consume significant computational resources and are prone to generating hallucinated content from Humans. In this work, we introduce $\texttt{DRAG}$, a novel framework for distilling RAG knowledge from large-scale Language Models (LLMs) into small LMs (SLMs). Our approach leverages evidence- and knowledge graph-based distillation, ensuring that the distilled model retains critical factual knowledge while significantly reducing model size and computational cost. By aligning the smaller model's predictions with a structured knowledge graph and ranked evidence, $\texttt{DRAG}$ effectively mitigates hallucinations and improves factual accuracy. We further present a case demonstrating how our framework mitigates user privacy risks and introduce a corresponding benchmark. Experimental evaluations on multiple benchmarks demonstrate that our method outperforms the prior competitive RAG methods like MiniRAG for SLMs by up to 27.7% using the same models, preserving high-level efficiency and reliability. With $\texttt{DRAG}$, we provide a practical and resource-efficient roadmap to deploying enhanced retrieval and generation capabilities in small-sized LLMs.

cs.CL↗

arXiv2025

Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents

CAPTCHAs have been a critical bottleneck for deploying web agents in real-world applications, often blocking them from completing end-to-end automation tasks. While modern multimodal LLM agents have demonstrated impressive performance in static perception tasks, their ability to handle interactive, multi-step reasoning challenges like CAPTCHAs is largely untested. To address this gap, we introduce Open CaptchaWorld, the first web-based benchmark and platform specifically designed to evaluate the visual reasoning and interaction capabilities of MLLM-powered agents through diverse and dynamic CAPTCHA puzzles. Our benchmark spans 20 modern CAPTCHA types, totaling 225 CAPTCHAs, annotated with a new metric we propose: CAPTCHA Reasoning Depth, which quantifies the number of cognitive and motor steps required to solve each puzzle. Experimental results show that humans consistently achieve near-perfect scores, state-of-the-art MLLM agents struggle significantly, with success rates at most 40.0% by Browser-Use Openai-o3, far below human-level performance, 93.3%. This highlights Open CaptchaWorld as a vital benchmark for diagnosing the limits of current multimodal agents and guiding the development of more robust multimodal reasoning systems. Code and Data are available at this https URL.

cs.AI↗

arXiv2025

Adaptive Mix for Semi-Supervised Medical Image Segmentation

Mix-up is a key technique for consistency regularization-based semi-supervised learning methods, blending two or more images to generate strong-perturbed samples for strong-weak pseudo supervision. Existing mix-up operations are performed either randomly or with predefined fixed rules, such as replacing low-confidence patches with high-confidence ones. The former lacks control over the perturbation degree, leading to overfitting on randomly perturbed samples, while the latter tends to generate images with trivial perturbations, both of which limit the effectiveness of consistency regularization. This paper aims to answer the following question: How can image mix-up perturbation be adaptively performed during training? To this end, we propose an Adaptive Mix algorithm (AdaMix) for image mix-up in a self-paced learning manner. Given that, in general, a model's performance gradually improves during training, AdaMix is equipped with a self-paced curriculum that, in the initial training stage, provides relatively simple perturbed samples and then gradually increases the difficulty of perturbed images by adaptively controlling the perturbation degree based on the model's learning state estimated by a self-paced regularize. We develop three frameworks with our AdaMix, i.e., AdaMix-ST, AdaMix-MT, and AdaMix-CT, for semi-supervised medical image segmentation. Extensive experiments on three public datasets show that the proposed frameworks can achieve superior performance. For example, compared with the state-of-the-art, AdaMix-CT achieves relative improvements of 2.62% in Dice similarity coefficient and 48.25% in average surface distance on the ACDC dataset with 10% labeled data. The results demonstrate that mix-up operations with dynamically adjusted perturbation strength based on the segmentation model's state can significantly enhance the effectiveness of consistency regularization.

cs.CV↗

arXiv2025

The Chinese pulsar timing array data release I. Polarimetry for 56 millisecond pulsars

We present polarization pulse profiles for 56 millisecond pulsars (MSPs) monitored by the Chinese Pulsar Timing Array (CPTA) collaboration using the Five-hundred-meter Aperture Spherical radio Telescope (FAST). The observations centered at 1.25 GHz with a raw bandwidth of 500 MHz. Due to the high sensitivity ($\sim$16 K/Jy) of the FAST telescope and our long integration time, the high signal-to-noise ratio polarization profiles show features hardly detected before. Among 56 pulsars, the polarization profiles of PSRs J0406$+$3039, J1327$+$3423, and J2022$+$2534 were not previously reported. 80\% of MSPs in the sample show weak components below 3\% of peak flux, 25\% of pulsars show interpulse-like structures, and most pulsars show linear polarization position angle jumps. Six pulsars seem to be emitting for full rotation phase, with another thirteen pulsars being good candidates for such a 360$^\circ$ radiator. We find that the distribution of the polarization percentage in our sample is compatible with the normal pulsar distribution. Our detailed evaluation of the MSP polarization properties suggests that the wave propagation effects in the pulsar magnetosphere are important in shaping the MSP polarization pulse profiles.

astro-ph.HE↗

arXiv2025

Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark

Rapid advancements in large language models (LLMs) have increased interest in deploying them on mobile devices for on-device AI applications. Mobile users interact differently with LLMs compared to desktop users, creating unique expectations and data biases. Current benchmark datasets primarily target at server and desktop environments, and there is a notable lack of extensive datasets specifically designed for mobile contexts. Additionally, mobile devices face strict limitations in storage and computing resources, constraining model size and capabilities, thus requiring optimized efficiency and prioritized knowledge. To address these challenges, we introduce Mobile-MMLU, a large-scale benchmark dataset tailored for mobile intelligence. It consists of 16,186 questions across 80 mobile-related fields, designed to evaluate LLM performance in realistic mobile scenarios. A challenging subset, Mobile-MMLU-Pro, provides advanced evaluation similar in size to MMLU-Pro but significantly more difficult than our standard full set. Both benchmarks use multiple-choice, order-invariant questions focused on practical mobile interactions, such as recipe suggestions, travel planning, and essential daily tasks. The dataset emphasizes critical mobile-specific metrics like inference latency, energy consumption, memory usage, and response quality, offering comprehensive insights into model performance under mobile constraints. Moreover, it prioritizes privacy and adaptability, assessing models' ability to perform on-device processing, maintain user privacy, and adapt to personalized usage patterns. Mobile-MMLU family offers a standardized framework for developing and comparing mobile-optimized LLMs, enabling advancements in productivity and decision-making within mobile computing environments. Our code and data are available at: https://github.com/VILA-Lab/Mobile-MMLU.

cs.CL↗

arXiv2025

Knowledge Consultation for Semi-Supervised Semantic Segmentation

Semi-Supervised Semantic Segmentation reduces reliance on extensive annotations by using unlabeled data and state-of-the-art models to improve overall performance. Despite the success of deep co-training methods, their underlying mechanisms remain underexplored. This work revisits Cross Pseudo Supervision with dual heterogeneous backbones and introduces Knowledge Consultation (SegKC) to further enhance segmentation performance. The proposed SegKC achieves significant improvements on Pascal and Cityscapes benchmarks, with mIoU scores of 87.1%, 89.2%, and 89.8% on Pascal VOC with the 1/4, 1/2, and full split partition, respectively, while maintaining a compact model architecture.

cs.CV↗

arXiv2025

Subclustering and Star Formation Efficiency in Three Protoclusters in the Central Molecular Zone

We present so far the highest resolution ($\sim$0.04") ALMA 1.3 mm continuum observations of three massive star-forming clumps in the Central Molecular Zone, namely 20 km s$^{-1}$ C1, 20 km $^{-1}$ C4, and Sgr C C4, which reveal prevalent compact millimeter emission. We extract the compact emission with $\textit{astrodendro}$ and identify a total of 199 fragments with a typical size of $\sim$370 AU, which represent the first sample of candidates of protostellar envelopes and disks and kernels of prestellar cores in these clumps that are likely forming star clusters. Compared with the protoclusters in the Galactic disk, the three protoclusters display a higher level of hierarchical clustering, likely a result of the stronger turbulence in the CMZ clumps. Compared with the mini-starbursts in the CMZ, Sgr B2 M and N, the three protoclusters also show stronger subclustering in conjunction with a lack of massive fragments. The efficiency of high-mass star formation of the three protoclusters is on average one order of magnitude lower than that of Sgr B2 M and N, despite a similar overall efficiency of converting gas into stars. The lower efficiency of high-mass star formation in the three protoclusters is likely attributed to hierarchical cluster formation.

astro-ph.GA↗

arXiv2025

Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization

Recent research indicates that large language models (LLMs) are susceptible to jailbreaking attacks that can generate harmful content. This paper introduces a novel token-level attack method, Adaptive Dense-to-Sparse Constrained Optimization (ADC), which has been shown to successfully jailbreak multiple open-source LLMs. Drawing inspiration from the difficulties of discrete token optimization, our method relaxes the discrete jailbreak optimization into a continuous optimization process while gradually increasing the sparsity of the optimizing vectors. This technique effectively bridges the gap between discrete and continuous space optimization. Experimental results demonstrate that our method is more effective and efficient than state-of-the-art token-level methods. On Harmbench, our approach achieves the highest attack success rate on seven out of eight LLMs compared to the latest jailbreak methods. Trigger Warning: This paper contains model behavior that can be offensive in nature.

cs.LG↗

arXiv2025

ALMA observations of massive clouds in the central molecular zone: slim filaments tracing parsec-scale shocks

The central molecular zone (CMZ) of our Galaxy exhibits widespread emission from SiO and various complex organic molecules (COMs), yet the exact origin of such emission is uncertain. Here we report the discovery of a unique class of long ($>$0.5 pc) and narrow ($<$0.03 pc) filaments in the emission of SiO 5$-$4 and eight additional molecular lines, including several COMs, in our ALMA 1.3 mm spectral line observations toward two massive molecular clouds in the CMZ, which we name as slim filaments. However, these filaments are not detected in the 1.3 mm continuum at the 5$σ$ level. Their line-of-sight velocities are coherent and inconsistent with being outflows. The column densities and relative abundances of the detected molecules are statistically similar to those in protostellar outflows but different from those in dense cores within the same clouds. Turbulent pressure in these filaments dominates over self gravity and leads to hydrostatic inequilibrium, indicating that they are a different class of objects than the dense gas filaments in dynamical equilibrium ubiquitously found in nearby molecular clouds. We argue that these newly detected slim filaments are associated with parsec-scale shocks, likely arising from dynamic interactions between shock waves and molecular clouds. The dissipation of the slim filaments may replenish SiO and COMs in the interstellar medium and lead to their widespread emission in the CMZ.

astro-ph.GA↗

arXiv2025

Elucidating the Design Space of Dataset Condensation

Dataset condensation, a concept within data-centric learning, efficiently transfers critical attributes from an original dataset to a synthetic version, maintaining both diversity and realism. This approach significantly improves model training efficiency and is adaptable across multiple application areas. Previous methods in dataset condensation have faced challenges: some incur high computational costs which limit scalability to larger datasets (e.g., MTT, DREAM, and TESLA), while others are restricted to less optimal design spaces, which could hinder potential improvements, especially in smaller datasets (e.g., SRe2L, G-VBSM, and RDED). To address these limitations, we propose a comprehensive design framework that includes specific, effective strategies like implementing soft category-aware matching and adjusting the learning rate schedule. These strategies are grounded in empirical evidence and theoretical backing. Our resulting approach, Elucidate Dataset Condensation (EDC), establishes a benchmark for both small and large-scale dataset condensation. In our testing, EDC achieves state-of-the-art accuracy, reaching 48.6% on ImageNet-1k with a ResNet-18 model at an IPC of 10, which corresponds to a compression ratio of 0.78%. This performance exceeds those of SRe2L, G-VBSM, and RDED by margins of 27.3%, 17.2%, and 6.6%, respectively.

cs.LG↗

arXiv2025

LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch

We detail the training of the LLM360 K2-65B model, scaling up our 360-degree OPEN SOURCE approach to the largest and most powerful models under project LLM360. While open-source LLMs continue to advance, the answer to "How are the largest LLMs trained?" remains unclear within the community. The implementation details for such high-capacity models are often protected due to business considerations associated with their high cost. This lack of transparency prevents LLM researchers from leveraging valuable insights from prior experience, e.g., "What are the best practices for addressing loss spikes?" The LLM360 K2 project addresses this gap by providing full transparency and access to resources accumulated during the training of LLMs at the largest scale. This report highlights key elements of the K2 project, including our first model, K2 DIAMOND, a 65 billion-parameter LLM that surpasses LLaMA-65B and rivals LLaMA2-70B, while requiring fewer FLOPs and tokens. We detail the implementation steps and present a longitudinal analysis of K2 DIAMOND's capabilities throughout its training process. We also outline ongoing projects such as TXT360, setting the stage for future models in the series. By offering previously unavailable resources, the K2 project also resonates with the 360-degree OPEN SOURCE principles of transparency, reproducibility, and accessibility, which we believe are vital in the era of resource-intensive AI research.

cs.LG↗

arXiv2024

ALMA Observations of Massive Clouds in the Central Molecular Zone: External-Pressure-Confined Dense Cores and Salpeter-like Core Mass Functions

We present Atacama Large Millimeter/submillimeter Array (ALMA) Band 6 (1.3 mm) observations of dense cores in three massive molecular clouds within the Central Molecular Zone (CMZ) of the Milky Way, including the Dust Ridge cloud e, Sgr C, and the 20 km s-1 cloud, at a spatial resolution of 2000 au. Among the 834 cores identified from the 1.3 mm continuum, we constrain temperatures and linewidths of 253 cores using local thermodynamic equilibrium (LTE) methods to fit the H2CO and/or CH3CN spectra. We determine their masses using the 1.3 mm dust continuum and derived temperatures, and then evaluate their virial parameters using the H2CO and/or CH3CN linewidths and construct the core mass functions (CMFs). We find that the contribution of external pressure is crucial for the virial equilibrium of the dense cores in the three clouds, which contrasts with the environment in the Galactic disk where dense cores are already bound even without the contribution of external pressure. We also find that the CMFs show a Salpeter-like slope in the high-mass (>~3-6 Msun) end, a change from previous works with our new temperature estimates. Combined with the possible top-heavy initial mass functions (IMFs) in the CMZ, our result suggests that gas accretion and further fragmentation may play important roles in transforming the CMF to the IMF.

astro-ph.GA↗

arXiv2024

Dataset Distillation via Curriculum Data Synthesis in Large Data Era

Dataset distillation or condensation aims to generate a smaller but representative subset from a large dataset, which allows a model to be trained more efficiently, meanwhile evaluating on the original testing data distribution to achieve decent performance. Previous decoupled methods like SRe$^2$L simply use a unified gradient update scheme for synthesizing data from Gaussian noise, while, we notice that the initial several update iterations will determine the final outline of synthesis, thus an improper gradient update strategy may dramatically affect the final generation quality. To address this, we introduce a simple yet effective global-to-local gradient refinement approach enabled by curriculum data augmentation ($\texttt{CDA}$) during data synthesis. The proposed framework achieves the current published highest accuracy on both large-scale ImageNet-1K and 21K with 63.2% under IPC (Images Per Class) 50 and 36.1% under IPC 20, using a regular input resolution of 224$\times$224 with faster convergence speed and less synthetic time. The proposed model outperforms the current state-of-the-art methods like SRe$^2$L, TESLA, and MTT by more than 4% Top-1 accuracy on ImageNet-1K/21K and for the first time, reduces the gap to its full-data training counterparts to less than absolute 15%. Moreover, this work represents the inaugural success in dataset distillation on the larger-scale ImageNet-21K dataset under the standard 224$\times$224 resolution. Our code and distilled ImageNet-21K dataset of 20 IPC, 2K recovery budget are available at https://github.com/VILA-Lab/SRe2L/tree/main/CDA.

cs.CV↗