arXiv subjects

Xue Song

Publications and source records attributed to Xue Song.

14 recordsLinked to original sources

arXiv2026

DECODE: Tackling Representation and Decision Degradation in Continual AI-Generated Image Detection

As generative models continue to evolve, AI-generated image detectors must incrementally adapt to emerging generative domains while preserving knowledge acquired from previous ones. This continual learning setting is particularly challenging because forensic traces are often subtle and generator-specific, making detectors highly vulnerable to catastrophic forgetting. Existing methods primarily address this problem by stabilizing feature representations, implicitly treating forgetting as a representation-level issue. In this paper, we show that this perspective is incomplete. We demonstrate that even when feature representations remain discriminative, the decision boundary can progressively drift as the classification head is continually optimized on new domains. These two effects jointly give rise to a compound failure mode, termed Dual Degradation. To overcome this challenge, we propose DECODE, a decoupled continual detection framework that jointly mitigates representation- and decision-level forgetting. Specifically, we introduce Subspace Diversity Regularization (SDR) to preserve diverse forensic representations and Closed-Form Decision Alignment (CDA) to recalibrate the shared classification head after each adapter merge without manual hyperparameter tuning. Extensive experiments on 19 generative domains show that DECODE achieves an average accuracy of 99.36% with only 0.39% forgetting, while further generalizing to 11 unseen generators with 95.36% accuracy.

cs.CV↗

arXiv2025

WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation

Recent advances in unified multimodal models (UMMs) have enabled impressive progress in visual comprehension and generation. However, existing datasets and benchmarks focus primarily on single-turn interactions, failing to capture the multi-turn, context-dependent nature of real-world image creation and editing. To address this gap, we present WEAVE, the first suite for in-context interleaved cross-modality comprehension and generation. Our suite consists of two complementary parts. WEAVE-100k is a large-scale dataset of 100K interleaved samples spanning over 370K dialogue turns and 500K images, covering comprehension, editing, and generation tasks that require reasoning over historical context. WEAVEBench is a human-annotated benchmark with 100 tasks based on 480 images, featuring a hybrid VLM judger evaluation framework based on both the reference image and the combination of the original image with editing instructions that assesses models' abilities in multi-turn generation, visual memory, and world-knowledge reasoning across diverse domains. Experiments demonstrate that training on WEAVE-100k enables vision comprehension, image editing, and comprehension-generation collaboration capabilities. Furthermore, it facilitates UMMs to develop emergent visual-memory capabilities, while extensive evaluations on WEAVEBench expose the persistent limitations and challenges of current approaches in multi-turn, context-aware image generation and editing. We believe WEAVE provides a view and foundation for studying in-context interleaved comprehension and generation for multi-modal community.

cs.CV↗

arXiv2025

Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning

We completely discard the conventional spatial prior in image representation and introduce a novel discrete visual tokenizer: Self-consistency Tokenizer (Selftok). At its design core, we compose an autoregressive (AR) prior -- mirroring the causal structure of language -- into visual tokens by using the reverse diffusion process of image generation. The AR property makes Selftok fundamentally distinct from traditional spatial tokens in the following two key ways: - Selftok offers an elegant and minimalist approach to unify diffusion and AR for vision-language models (VLMs): By representing images with Selftok tokens, we can train a VLM using a purely discrete autoregressive architecture -- like that in LLMs -- without requiring additional modules or training objectives. - We theoretically show that the AR prior satisfies the Bellman equation, whereas the spatial prior does not. Therefore, Selftok supports reinforcement learning (RL) for visual generation with effectiveness comparable to that achieved in LLMs. Besides the AR property, Selftok is also a SoTA tokenizer that achieves a favorable trade-off between high-quality reconstruction and compression rate. We use Selftok to build a pure AR VLM for both visual comprehension and generation tasks. Impressively, without using any text-image training pairs, a simple policy gradient RL working in the visual tokens can significantly boost the visual generation benchmark, surpassing all the existing models by a large margin. Therefore, we believe that Selftok effectively addresses the long-standing challenge that visual tokens cannot support effective RL. When combined with the well-established strengths of RL in LLMs, this brings us one step closer to realizing a truly multimodal LLM. Project Page: https://selftok-team.github.io/report/.

cs.CV↗

arXiv2025

Revealing the Implicit Noise-based Imprint of Generative Models

With the rapid advancement of vision generation models, the potential security risks stemming from synthetic visual content have garnered increasing attention, posing significant challenges for AI-generated image detection. Existing methods suffer from inadequate generalization capabilities, resulting in unsatisfactory performance on emerging generative models. To address this issue, this paper presents NIRNet (Noise-based Imprint Revealing Network), a novel framework that leverages noise-based imprint for the detection task. Specifically, we propose a novel Noise-based Imprint Simulator to capture intrinsic patterns imprinted in images generated by different models. By aggregating imprint from various generative models, imprint of future models can be extrapolated to expand training data, thereby enhancing generalization and robustness. Furthermore, we design a new pipeline that pioneers the use of noise patterns, derived from a Noise-based Imprint Extractor, alongside other visual features for AI-generated image detection, significantly improving detection performance. Our approach achieves state-of-the-art performance across seven diverse benchmarks, including five public datasets and two newly proposed generalization tests, demonstrating its superior generalization and effectiveness. Paper Submission: pdf

cs.CV↗

arXiv2024

StyleStudio: Text-Driven Style Transfer with Selective Control of Style Elements

Text-driven style transfer aims to merge the style of a reference image with content described by a text prompt. Recent advancements in text-to-image models have improved the nuance of style transformations, yet significant challenges remain, particularly with overfitting to reference styles, limiting stylistic control, and misaligning with textual content. In this paper, we propose three complementary strategies to address these issues. First, we introduce a cross-modal Adaptive Instance Normalization (AdaIN) mechanism for better integration of style and text features, enhancing alignment. Second, we develop a Style-based Classifier-Free Guidance (SCFG) approach that enables selective control over stylistic elements, reducing irrelevant influences. Finally, we incorporate a teacher model during early generation stages to stabilize spatial layouts and mitigate artifacts. Our extensive evaluations demonstrate significant improvements in style transfer quality and alignment with textual prompts. Furthermore, our approach can be integrated into existing style transfer frameworks without fine-tuning.

cs.CV↗

arXiv2024

The Simons Observatory: Design, Integration, and Current Status of Small Aperture Telescopes

The Simons Observatory (SO) is a cosmic microwave background (CMB) survey experiment located in the Atacama Desert in Chile at an elevation of 5200 meters, nominally consisting of an array of three 0.42-meter small aperture telescopes (SATs) and one 6-meter large aperture telescope (LAT). SO will make accurate measurements of the CMB temperature and polarization spanning six frequency bands ranging from 27 to 280 GHz, fielding a total of $\sim$68,000 detectors covering angular scales between one arcminute to tens of degrees. In this paper, we focus on the SATs, which are tailored to search for primordial gravitational waves, with the primary science goal of measuring the primordial tensor-to-scalar ratio \textit{r} at a target level of $\sigma(r) \approx 0.003$. We discuss the design drivers, scientific impact, and current deployment status of the three SATs, which are scheduled to start taking data in the coming year. The SATs aim to map 10\% of the sky at a 2 $\mu$K-arcmin noise level observing at mid-frequencies (93/145 GHz), with additional ultra-high-frequency (225/280 GHz) and low-frequency (27/39 GHz) targets to yield galactic foreground-subtracted measurements.

astro-ph.IM↗

arXiv2024

LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Pair

In this paper, we propose the LoRA of Change (LoC) framework for image editing with visual instructions, i.e., before-after image pairs. Compared to the ambiguities, insufficient specificity, and diverse interpretations of natural language, visual instructions can accurately reflect users' intent. Building on the success of LoRA in text-based image editing and generation, we dynamically learn an instruction-specific LoRA to encode the "change" in a before-after image pair, enhancing the interpretability and reusability of our model. Furthermore, generalizable models for image editing with visual instructions typically require quad data, i.e., a before-after image pair, along with query and target images. Due to the scarcity of such quad data, existing models are limited to a narrow range of visual instructions. To overcome this limitation, we introduce the LoRA Reverse optimization technique, enabling large-scale training with paired data alone. Extensive qualitative and quantitative experiments demonstrate that our model produces high-quality images that align with user intent and support a broad spectrum of real-world visual instructions.

cs.CV↗

arXiv2024

Doubly Abductive Counterfactual Inference for Text-based Image Editing

We study text-based image editing (TBIE) of a single image by counterfactual inference because it is an elegant formulation to precisely address the requirement: the edited image should retain the fidelity of the original one. Through the lens of the formulation, we find that the crux of TBIE is that existing techniques hardly achieve a good trade-off between editability and fidelity, mainly due to the overfitting of the single-image fine-tuning. To this end, we propose a Doubly Abductive Counterfactual inference framework (DAC). We first parameterize an exogenous variable as a UNet LoRA, whose abduction can encode all the image details. Second, we abduct another exogenous variable parameterized by a text encoder LoRA, which recovers the lost editability caused by the overfitted first abduction. Thanks to the second abduction, which exclusively encodes the visual transition from post-edit to pre-edit, its inversion -- subtracting the LoRA -- effectively reverts pre-edit back to post-edit, thereby accomplishing the edit. Through extensive experiments, our DAC achieves a good trade-off between editability and fidelity. Thus, we can support a wide spectrum of user editing intents, including addition, removal, manipulation, replacement, style transfer, and facial change, which are extensively validated in both qualitative and quantitative evaluations. Codes are in https://github.com/xuesong39/DAC.

cs.CV↗

arXiv2024

Identity-Driven Multimedia Forgery Detection via Reference Assistance

Recent advancements in "deepfake" techniques have paved the way for generating various media forgeries. In response to the potential hazards of these media forgeries, many researchers engage in exploring detection methods, increasing the demand for high-quality media forgery datasets. Despite this, existing datasets have certain limitations. Firstly, most datasets focus on manipulating visual modality and usually lack diversity, as only a few forgery approaches are considered. Secondly, the quality of media is often inadequate in clarity and naturalness. Meanwhile, the size of the dataset is also limited. Thirdly, it is commonly observed that real-world forgeries are motivated by identity, yet the identity information of the individuals portrayed in these forgeries within existing datasets remains under-explored. For detection, identity information could be an essential clue to boost performance. Moreover, official media concerning relevant identities on the Internet can serve as prior knowledge, aiding both the audience and forgery detectors in determining the true identity. Therefore, we propose an identity-driven multimedia forgery dataset, IDForge, which contains 249,138 video shots sourced from 324 wild videos of 54 celebrities collected from the Internet. The fake video shots involve 9 types of manipulation across visual, audio, and textual modalities. Additionally, IDForge provides extra 214,438 real video shots as a reference set for the 54 celebrities. Correspondingly, we propose the Reference-assisted Multimodal Forgery Detection Network (R-MFDN), aiming at the detection of deepfake videos. Through extensive experiments on the proposed dataset, we demonstrate the effectiveness of R-MFDN on the multimedia detection task.

cs.MM↗

arXiv2023

Transferability Estimation Based On Principal Gradient Expectation

Transfer learning aims to improve the performance of target tasks by transferring knowledge acquired in source tasks. The standard approach is pre-training followed by fine-tuning or linear probing. Especially, selecting a proper source domain for a specific target domain under predefined tasks is crucial for improving efficiency and effectiveness. It is conventional to solve this problem via estimating transferability. However, existing methods can not reach a trade-off between performance and cost. To comprehensively evaluate estimation methods, we summarize three properties: stability, reliability and efficiency. Building upon them, we propose Principal Gradient Expectation(PGE), a simple yet effective method for assessing transferability. Specifically, we calculate the gradient over each weight unit multiple times with a restart scheme, and then we compute the expectation of all gradients. Finally, the transferability between the source and target is estimated by computing the gap of normalized principal gradients. Extensive experiments show that the proposed metric is superior to state-of-the-art methods on all properties.

cs.CV↗

arXiv2022

Text-driven Video Prediction

Current video generation models usually convert signals indicating appearance and motion received from inputs (e.g., image, text) or latent spaces (e.g., noise vectors) into consecutive frames, fulfilling a stochastic generation process for the uncertainty introduced by latent code sampling. However, this generation pattern lacks deterministic constraints for both appearance and motion, leading to uncontrollable and undesirable outcomes. To this end, we propose a new task called Text-driven Video Prediction (TVP). Taking the first frame and text caption as inputs, this task aims to synthesize the following frames. Specifically, appearance and motion components are provided by the image and caption separately. The key to addressing the TVP task depends on fully exploring the underlying motion information in text descriptions, thus facilitating plausible video generation. In fact, this task is intrinsically a cause-and-effect problem, as the text content directly influences the motion changes of frames. To investigate the capability of text in causal inference for progressive motion information, our TVP framework contains a Text Inference Module (TIM), producing step-wise embeddings to regulate motion inference for subsequent frames. In particular, a refinement mechanism incorporating global motion semantics guarantees coherent generation. Extensive experiments are conducted on Something-Something V2 and Single Moving MNIST datasets. Experimental results demonstrate that our model achieves better results over other baselines, verifying the effectiveness of the proposed framework.

cs.CV↗

arXiv2018

The CST Bounce Universe model -- a parametric study

A bounce universe model with a scale-invariant and stable spectrum of primordial density perturbations was constructed using a consistent truncation of the D-brane dynamics from Type IIB string theory. A coupling was introduced between the tachyon field and the adjoint Higgs field on the D3-branes to lock the tachyon at the top of its potential hill and to model the bounce process, which is known as the Coupled Scalar and Tachyon Bounce (CSTB) Universe. The CSTB model has been shown to be ghost free, and it fulfils the null energy condition; in addition, it can also solve the Big Bang cosmic singularity problem. In this paper we conduct an extensive follow-up study of the parameter space of the CSTB model. In particular we are interested in the parameter values that can produce a single bounce to arrive at a radiation-dominated universe. We further establish that the CSTB universe is a viable alternative to inflation, as it can naturally produce a sufficient number of e-foldings in the locked inflation epoch and in the post-bounce expansion to overcome the four fundamental limitations of the Big Bang cosmology, which are flatness, horizon, homogeneity and singularity, resulting in a universe of the current size.

gr-qc↗

arXiv2016

Design of 280 GHz feedhorn-coupled TES arrays for the balloon-borne polarimeter SPIDER

We describe 280 GHz bolometric detector arrays that instrument the balloon-borne polarimeter SPIDER. A primary science goal of SPIDER is to measure the large-scale B-mode polarization of the cosmic microwave background in search of the cosmic-inflation, gravitational-wave signature. 280 GHz channels aid this science goal by constraining the level of B-mode contamination from galactic dust emission. We present the focal plane unit design, which consists of a 16$\times$16 array of conical, corrugated feedhorns coupled to a monolithic detector array fabricated on a 150 mm diameter silicon wafer. Detector arrays are capable of polarimetric sensing via waveguide probe-coupling to a multiplexed array of transition-edge-sensor (TES) bolometers. The SPIDER receiver has three focal plane units at 280 GHz, which in total contains 765 spatial pixels and 1,530 polarization sensitive bolometers. By fabrication and measurement of single feedhorns, we demonstrate 14.7$^{\circ}$ FHWM Gaussian-shaped beams with $<$1% ellipticity in a 30% fractional bandwidth centered at 280 GHz. We present electromagnetic simulations of the detection circuit, which show 94% band-averaged, single-polarization coupling efficiency, 3% reflection and 3% radiative loss. Lastly, we demonstrate a low thermal conductance bolometer, which is well-described by a simple TES model and exhibits an electrical noise equivalent power (NEP) = 2.6 $\times$ 10$^{-17}$ W/$\sqrt{\mathrm{Hz}}$, consistent with the phonon noise prediction.

astro-ph.IM↗

arXiv2014

Time structure measurement of the storage ring with the time-resolved X-ray excited optical luminescence method at SSRF

Measuring the time structure of the storage ring on the sample spot inside the experimental hutch is a foundational step during the time-resolved experiments using the pulsed synchrotron X-rays with the time structure defined by the storage ring. In this work, the method of time-resolved X-ray excited optical luminescence was designed and implemented to do the measurement. This method is based on the principle of time-correlated single photon counting techniques. The measurement system consists of a spectrometer with a detector of photomultiplier tube, a timing system, a set of Nuclear Instrument Modules and a luminescent material of Zinc Oxide. The measurement was performed on the X-ray absorbed fine structure spectrum beamline at Shanghai Synchrotron Radiation Facility. The results show that this system can be used to measure the time structure of the storage ring with a precision of less than 1 ns.

physics.ins-det↗