arXiv subjects

Kaiwen Zheng

Publications and source records attributed to Kaiwen Zheng.

At least 55 records · Page 3Linked to original sources

arXiv2025

Visual Generation Without Guidance

Classifier-Free Guidance (CFG) has been a default technique in various visual generative models, yet it requires inference from both conditional and unconditional models during sampling. We propose to build visual models that are free from guided sampling. The resulting algorithm, Guidance-Free Training (GFT), matches the performance of CFG while reducing sampling to a single model, halving the computational cost. Unlike previous distillation-based approaches that rely on pretrained CFG networks, GFT enables training directly from scratch. GFT is simple to implement. It retains the same maximum likelihood objective as CFG and differs mainly in the parameterization of conditional models. Implementing GFT requires only minimal modifications to existing codebases, as most design choices and hyperparameters are directly inherited from CFG. Our extensive experiments across five distinct visual models demonstrate the effectiveness and versatility of GFT. Across domains of diffusion, autoregressive, and masked-prediction modeling, GFT consistently achieves comparable or even lower FID scores, with similar diversity-fidelity trade-offs compared with CFG baselines, all while being guidance-free. Code will be available at https://github.com/thu-ml/GFT.

cs.CV↗

arXiv2025

Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator

While likelihood-based generative models, particularly diffusion and autoregressive models, have achieved remarkable fidelity in visual generation, the maximum likelihood estimation (MLE) objective, which minimizes the forward KL divergence, inherently suffers from a mode-covering tendency that limits the generation quality under limited model capacity. In this work, we propose Direct Discriminative Optimization (DDO) as a unified framework that integrates likelihood-based generative training and GAN-type discrimination to bypass this fundamental constraint by exploiting reverse KL and self-generated negative signals. Our key insight is to parameterize a discriminator implicitly using the likelihood ratio between a learnable target model and a fixed reference model, drawing parallels with the philosophy of Direct Preference Optimization (DPO). Unlike GANs, this parameterization eliminates the need for joint training of generator and discriminator networks, allowing for direct, efficient, and effective finetuning of a well-trained model to its full potential beyond the limits of MLE. DDO can be performed iteratively in a self-play manner for progressive model refinement, with each round requiring less than 1% of pretraining epochs. Our experiments demonstrate the effectiveness of DDO by significantly advancing the previous SOTA diffusion model EDM, reducing FID scores from 1.79/1.58/1.96 to new records of 1.30/0.97/1.26 on CIFAR-10/ImageNet-64/ImageNet 512x512 datasets without any guidance mechanisms, and by consistently improving both guidance-free and CFG-enhanced FIDs of visual autoregressive models on ImageNet 256x256.

cs.CV↗

arXiv2025

Diffusion Bridge Implicit Models

Denoising diffusion bridge models (DDBMs) are a powerful variant of diffusion models for interpolating between two arbitrary paired distributions given as endpoints. Despite their promising performance in tasks like image translation, DDBMs require a computationally intensive sampling process that involves the simulation of a (stochastic) differential equation through hundreds of network evaluations. In this work, we take the first step in fast sampling of DDBMs without extra training, motivated by the well-established recipes in diffusion models. We generalize DDBMs via a class of non-Markovian diffusion bridges defined on the discretized timesteps concerning sampling, which share the same marginal distributions and training objectives, give rise to generative processes ranging from stochastic to deterministic, and result in diffusion bridge implicit models (DBIMs). DBIMs are not only up to 25$\times$ faster than the vanilla sampler of DDBMs but also induce a novel, simple, and insightful form of ordinary differential equation (ODE) which inspires high-order numerical solvers. Moreover, DBIMs maintain the generation diversity in a distinguished way, by using a booting noise in the initial sampling step, which enables faithful encoding, reconstruction, and semantic interpolation in image translation tasks. Code is available at https://github.com/thu-ml/DiffusionBridge.

cs.LG↗

arXiv2025

Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling

Masked diffusion models (MDMs) have emerged as a popular research topic for generative modeling of discrete data, thanks to their superior performance over other discrete diffusion models, and are rivaling the auto-regressive models (ARMs) for language modeling tasks. The recent effort in simplifying the masked diffusion framework further leads to alignment with continuous-space diffusion models and more principled training and sampling recipes. In this paper, however, we reveal that both training and sampling of MDMs are theoretically free from the time variable, arguably the key signature of diffusion models, and are instead equivalent to masked models. The connection on the sampling aspect is drawn by our proposed first-hitting sampler (FHS). Specifically, we show that the FHS is theoretically equivalent to MDMs' original generation process while significantly alleviating the time-consuming categorical sampling and achieving a 20$\times$ speedup. In addition, our investigation raises doubts about whether MDMs can truly beat ARMs in text generation. We identify, for the first time, an underlying numerical issue, even with the commonly used 32-bit floating-point precision, which results in inaccurate categorical sampling. We show that it lowers the effective temperature both theoretically and empirically, and the resulting decrease in token diversity makes previous evaluations, which assess the generation quality solely through the incomplete generative perplexity metric, somewhat unfair.

cs.LG↗

arXiv2025

Elucidating the Preconditioning in Consistency Distillation

Consistency distillation is a prevalent way for accelerating diffusion models adopted in consistency (trajectory) models, in which a student model is trained to traverse backward on the probability flow (PF) ordinary differential equation (ODE) trajectory determined by the teacher model. Preconditioning is a vital technique for stabilizing consistency distillation, by linear combining the input data and the network output with pre-defined coefficients as the consistency function. It imposes the boundary condition of consistency functions without restricting the form and expressiveness of the neural network. However, previous preconditionings are hand-crafted and may be suboptimal choices. In this work, we offer the first theoretical insights into the preconditioning in consistency distillation, by elucidating its design criteria and the connection to the teacher ODE trajectory. Based on these analyses, we further propose a principled way dubbed \textit{Analytic-Precond} to analytically optimize the preconditioning according to the consistency gap (defined as the gap between the teacher denoiser and the optimal student denoiser) on a generalized teacher ODE. We demonstrate that Analytic-Precond can facilitate the learning of trajectory jumpers, enhance the alignment of the student trajectory with the teacher's, and achieve $2\times$ to $3\times$ training acceleration of consistency trajectory models in multi-step generation across various datasets.

cs.LG↗

arXiv2025

Multimodal Representation Learning Techniques for Comprehensive Facial State Analysis

Multimodal foundation models have significantly improved feature representation by integrating information from multiple modalities, making them highly suitable for a broader set of applications. However, the exploration of multimodal facial representation for understanding perception has been limited. Understanding and analyzing facial states, such as Action Units (AUs) and emotions, require a comprehensive and robust framework that bridges visual and linguistic modalities. In this paper, we present a comprehensive pipeline for multimodal facial state analysis. First, we compile a new Multimodal Face Dataset (MFA) by generating detailed multilevel language descriptions of face, incorporating Action Unit (AU) and emotion descriptions, by leveraging GPT-4o. Second, we introduce a novel Multilevel Multimodal Face Foundation model (MF^2) tailored for Action Unit (AU) and emotion recognition. Our model incorporates comprehensive visual feature modeling at both local and global levels of face image, enhancing its ability to represent detailed facial appearances. This design aligns visual representations with structured AU and emotion descriptions, ensuring effective cross-modal integration. Third, we develop a Decoupled Fine-Tuning Network (DFN) that efficiently adapts MF^2 across various tasks and datasets. This approach not only reduces computational overhead but also broadens the applicability of the foundation model to diverse scenarios. Experimentation show superior performance for AU and emotion detection tasks.

cs.CV↗

arXiv2025

Surface Morphology Assisted Trapping of Strongly Coupled Electron-on-Neon Charge States

Single electrons confined to a free neon surface and manipulated through the circuit quantum electrodynamics (circuit QED) architecture is a promising novel quantum computing platform. Understanding the exact physical nature of the electron-on-neon (eNe) charge states is important for realizing this platform's potential for quantum technologies. We investigate how resonator trench depth and substrate surface properties influence the formation of eNe charge states and their coupling to microwave resonators. Through experimental observation supported by modeling, we find that shallow-depth etching of the resonator features maximizes coupling strength. By comparing the trapping statistics and surface morphology of devices with altered trench roughness, our work reveals the role of fabrication-induced surface features in the formation of strongly coupled eNe states.

quant-ph↗

arXiv2025

Simons Observatory: Characterization of the Large Aperture Telescope Receiver

The Simons Observatory (SO) is a ground-based cosmic microwave background (CMB) survey experiment that currently consists of three 0.42m small-aperture telescopes (SATs) and one 6m large-aperture telescope (LAT), located at an elevation of 5200m in the Atacama Desert in Chile. At the LAT's focal plane, SO will install >62,000 transition-edge sensor detectors across 13 optics tubes (OTs) within the Large Aperture Telescope Receiver (LATR), the largest cryogenic camera ever built to observe the CMB. Here we report on the validation of the LATR in the laboratory and the subsequent dark testing and validation within the LAT. We show that the LATR meets cryogenic, optical, and detector specifications required for high-sensitivity measurements of the CMB. At the time of writing, the LATR is installed in the LAT with six OTs (corresponding to >31,000 detectors), and the LAT mirrors and remaining seven OTs are undergoing development.

astro-ph.IM↗

arXiv2024

The Simons Observatory: Design, Optimization, and Performance of Low Frequency Detectors

The Simons Observatory (SO) is a cosmic microwave background (CMB) experiment located in the Atacama Desert in Chile that will make precise temperature and polarization measurements over six spectral bands ranging from 27 to 285 GHz. Three small aperture telescopes (SATs) and one large aperture telescope (LAT) will house $\sim$60,000 detectors and cover angular scales between one arcminute and tens of degrees. We present the performance of the dichroic, low-frequency (LF) lenslet-coupled sinuous antenna transition-edge sensor (TES) bolometer arrays with bands centered at 27 and 39 GHz. The LF focal plane will primarily characterize Galactic synchrotron emission as a critical part of foreground subtraction from CMB data. We will discuss the design, optimization, and current testing status of these pixels.

astro-ph.IM↗

arXiv2024

Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model

Diffusion models have obtained substantial progress in image-to-video generation. However, in this paper, we find that these models tend to generate videos with less motion than expected. We attribute this to the issue called conditional image leakage, where the image-to-video diffusion models (I2V-DMs) tend to over-rely on the conditional image at large time steps. We further address this challenge from both inference and training aspects. First, we propose to start the generation process from an earlier time step to avoid the unreliable large-time steps of I2V-DMs, as well as an initial noise distribution with optimal analytic expressions (Analytic-Init) by minimizing the KL divergence between it and the actual marginal distribution to bridge the training-inference gap. Second, we design a time-dependent noise distribution (TimeNoise) for the conditional image during training, applying higher noise levels at larger time steps to disrupt it and reduce the model's dependency on it. We validate these general strategies on various I2V-DMs on our collected open-domain image benchmark and the UCF101 dataset. Extensive results show that our methods outperform baselines by producing higher motion scores with lower errors while maintaining image alignment and temporal consistency, thereby yielding superior overall performance and enabling more accurate motion control. The project page: \url{https://cond-image-leak.github.io/}.

cs.CV↗

arXiv2024

Consistency Diffusion Bridge Models

Diffusion models (DMs) have become the dominant paradigm of generative modeling in a variety of domains by learning stochastic processes from noise to data. Recently, diffusion denoising bridge models (DDBMs), a new formulation of generative modeling that builds stochastic processes between fixed data endpoints based on a reference diffusion process, have achieved empirical success across tasks with coupled data distribution, such as image-to-image translation. However, DDBM's sampling process typically requires hundreds of network evaluations to achieve decent performance, which may impede their practical deployment due to high computational demands. In this work, inspired by the recent advance of consistency models in DMs, we tackle this problem by learning the consistency function of the probability-flow ordinary differential equation (PF-ODE) of DDBMs, which directly predicts the solution at a starting step given any point on the ODE trajectory. Based on a dedicated general-form ODE solver, we propose two paradigms: consistency bridge distillation and consistency bridge training, which is flexible to apply on DDBMs with broad design choices. Experimental results show that our proposed method could sample $4\times$ to $50\times$ faster than the base DDBM and produce better visual quality given the same step in various tasks with pixel resolution ranging from $64 \times 64$ to $256 \times 256$, as well as supporting downstream tasks such as semantic interpolation in the data space.

cs.LG↗

arXiv2024

Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control

Drawing upon recent advances in language model alignment, we formulate offline Reinforcement Learning as a two-stage optimization problem: First pretraining expressive generative policies on reward-free behavior datasets, then fine-tuning these policies to align with task-specific annotations like Q-values. This strategy allows us to leverage abundant and diverse behavior data to enhance generalization and enable rapid adaptation to downstream tasks using minimal annotations. In particular, we introduce Efficient Diffusion Alignment (EDA) for solving continuous control problems. EDA utilizes diffusion models for behavior modeling. However, unlike previous approaches, we represent diffusion policies as the derivative of a scalar neural network with respect to action inputs. This representation is critical because it enables direct density calculation for diffusion models, making them compatible with existing LLM alignment theories. During policy fine-tuning, we extend preference-based alignment methods like Direct Preference Optimization (DPO) to align diffusion behaviors with continuous Q-functions. Our evaluation on the D4RL benchmark shows that EDA exceeds all baseline methods in overall performance. Notably, EDA maintains about 95\% of performance and still outperforms several baselines given only 1\% of Q-labelled data during fine-tuning.

cs.LG↗

arXiv2024

The Simons Observatory: A Minimum-Cost Matching Algorithm for Pairing Measured Resonances with Designed Detectors

The Simons Observatory (SO) is a ground-based cosmic microwave background experiment currently being deployed to Cerro Toco in the Atacama Desert of Chile. The initial deployment of SO, consisting of three 0.46m-diameter small-aperture telescopes and one 6m-primary large-aperture telescope, will field over 60,000 transition-edge sensors that will observe at frequencies between 30 GHz and 280 GHz. SO will read out its detectors using Superconducting Quantum Interference Device (SQUID) microwave-frequency multiplexing $μ$mux, a form of frequency division multiplexing where an RF-SQUID couples each TES bolometer to a superconducting resonator tuned to a unique frequency. Resonator frequencies are spaced roughly every 2 MHz between 4 and 6 GHz, allowing for multiplexing factors on the order of 1000. One challenge of $μ$mux is matching each tracked resonator with its corresponding physical detector. Variations in resonator fabrication, and frequency shifts between cooldowns caused by trapped flux can cause the measured resonance frequencies to deviate significantly from their designed values. In this study, we introduce a method for pairing measured and designed resonators by constructing a bipartite graph based on the two resonator sets, and assigning edge weights based on measured resonator and detector properties such as resonance frequency, detector pointing, and assigned bias lines. Finding the minimum-cost matching for a given set of edge weights is a well-studied problem that can be solved very quickly, and this matching tells us the best assignment of measured resonators to designed detectors for our input parameters. We will present results based on the first on-sky measurements from SAT1, the first SO MF small-aperture telescope.

astro-ph.IM↗

arXiv2024

The Simons Observatory: Dark Characterization of the Large Aperture Telescope

The Simons Observatory (SO) is a cosmic microwave background experiment composed of three 0.42 m Small Aperture Telescopes (SATs) and one 6 m Large Aperture Telescope (LAT) in the Atacama Desert of Chile. The Large Aperture Telescope Receiver (LATR) was integrated into the LAT in August 2023; however, because mirrors were not yet installed, the LATR optical chain was capped at the 4K stage. In this dark configuration we are able to characterize many elements of the instrument without contributions from atmospheric noise. Here we show this noise is below the required upper limit and its features are well described with a simple noise model. Maps produced using this noise model have properties that are in good agreement with the white noise levels of our dark data. Additionally, we show that our nominal scan strategy has a minimal effect on the noise when compared to the noise when the telescope is stationary

astro-ph.IM↗

arXiv2024

The Simons Observatory: Studies of Detector Yield and Readout Noise From the First Large-Scale Deployment of Microwave Multiplexing at the Large Aperture Telescope

The Simons Observatory is a new ground-based cosmic microwave background experiment, which is currently being commissioned in Chile's Atacama Desert. During its survey, the observatory's small aperture telescopes will map 10% of the sky in bands centered at frequencies ranging from 27 to 280 GHz to constrain cosmic inflation models, and its large aperture telescope will map 40% of the sky in the same bands to constrain cosmological parameters and use weak lensing to study large-scale structure. To achieve these science goals, the Simons Observatory is deploying these telescopes' receivers with 60,000 state-of-the-art superconducting transition-edge sensor bolometers for its first five year survey. Reading out this unprecedented number of cryogenic sensors, however, required the development of a novel readout system. The SMuRF electronics were developed to enable high-density readout of superconducting sensors using cryogenic microwave SQUID multiplexing technology. The commissioning of the SMuRF systems at the Simons Observatory is the largest deployment to date of microwave multiplexing technology for transition-edge sensors. In this paper, we show that a significant fraction of the systems deployed so far to the Simons Observatory's large aperture telescope meet baseline specifications for detector yield and readout noise in this early phase of commissioning.

astro-ph.IM↗

arXiv2024

Simons Observatory: Pre-deployment Performance of a Large Aperture Telescope Optics Tube in the 90 and 150 GHz Spectral Bands

The Simons Observatory will map the temperature and polarization over half of the sky, at millimeter wavelengths in six spectral bands from the Atacama Desert in Chile. These data will provide new insights into the genesis, content, and history of our Universe; the astrophysics of galaxies and galaxy clusters; objects in our solar system; and time-varying astrophysical phenomena. This ambitious new instrument suite, initially comprising three 0.5 m small-aperture telescopes and one 6 m large aperture telescope, is designed using a common combination of new technologies and new implementations to realize an observatory significantly more capable than the previous generation. In this paper, we present the pre-deployment performance of the first mid-frequency "optics tube" which will be fielded on the large aperture telescope with sensitivity to the 90 and 150 GHz spectral bands. This optics tube contains lenses, filters, detectors, and readout components, all of which operate at cryogenic temperatures. It is one of seven that form the core of the large aperture telescope receiver in its initial deployment. We describe this optics tube, including details of comprehensive testing methods, new techniques for beam and passband characterization, and its measured performance. The performance metrics include beams, optical efficiency, passbands, and forecasts for the on-sky performance of the system. We forecast a sensitivity that exceeds the requirements of the large aperture telescope with greater than 30% margin in each spectral band, and predict that the instrument will realize diffraction-limited performance and the expected detector passbands.

astro-ph.IM↗

arXiv2024

The Simons Observatory: Design, integration, and testing of the small aperture telescopes

The Simons Observatory (SO) is a cosmic microwave background (CMB) survey experiment that includes small-aperture telescopes (SATs) observing from an altitude of 5,200 m in the Atacama Desert in Chile. The SO SATs will cover six spectral bands between 27 and 280 GHz to search for primordial B-modes to a sensitivity of $σ(r)=0.002$, with quantified systematic errors well below this value. Each SAT is a self-contained cryogenic telescope with a 35$^\circ$ field of view, 42 cm diameter optical aperture, 40 K half-wave plate, 1 K refractive optics, and $<0.1$ K focal plane that holds $>12,000$ TES detectors. We describe the nominal design of the SATs and present details about the integration and testing for one operating at 93 and 145 GHz.

astro-ph.IM↗

arXiv2024

Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models

We introduce Vidu, a high-performance text-to-video generator that is capable of producing 1080p videos up to 16 seconds in a single generation. Vidu is a diffusion model with U-ViT as its backbone, which unlocks the scalability and the capability for handling long videos. Vidu exhibits strong coherence and dynamism, and is capable of generating both realistic and imaginative videos, as well as understanding some professional photography techniques, on par with Sora -- the most powerful reported text-to-video generator. Finally, we perform initial experiments on other controllable video generation, including canny-to-video generation, video prediction and subject-driven generation, which demonstrate promising results.

cs.CV↗