arXiv subjects

Han Cai

Publications and source records attributed to Han Cai.

At least 55 records · Page 3Linked to original sources

arXiv2025

Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code is open-sourced and is available at https://github.com/svg-project/Sparse-VideoGen

cs.CV↗

arXiv2025

Accelerate High-Quality Diffusion Models with Inner Loop Feedback

We propose Inner Loop Feedback (ILF), a novel approach to accelerate diffusion models' inference. ILF trains a lightweight module to predict future features in the denoising process by leveraging the outputs from a chosen diffusion backbone block at a given time step. This approach exploits two key intuitions; (1) the outputs of a given block at adjacent time steps are similar, and (2) performing partial computations for a step imposes a lower burden on the model than skipping the step entirely. Our method is highly flexible, since we find that the feedback module itself can simply be a block from the diffusion backbone, with all settings copied. Its influence on the diffusion forward can be tempered with a learnable scaling factor from zero initialization. We train this module using distillation losses; however, unlike some prior work where a full diffusion backbone serves as the student, our model freezes the backbone, training only the feedback module. While many efforts to optimize diffusion models focus on achieving acceptable image quality in extremely few steps (1-4 steps), our emphasis is on matching best case results (typically achieved in 20 steps) while significantly reducing runtime. ILF achieves this balance effectively, demonstrating strong performance for both class-to-image generation with diffusion transformer (DiT) and text-to-image generation with DiT-based PixArt-alpha and PixArt-sigma. The quality of ILF's 1.7x-1.8x speedups are confirmed by FID, CLIP score, CLIP Image Quality Assessment, ImageReward, and qualitative comparisons. Project information is available at https://mgwillia.github.io/ilf.

cs.CV↗

arXiv2025

COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training

FP8 training has emerged as a promising method for improving training efficiency. Existing frameworks accelerate training by applying FP8 computation to linear layers while leaving optimizer states and activations in higher precision, which fails to fully optimize memory usage. This paper introduces COAT (Compressing Optimizer States and Activations for FP8 Training), a novel FP8 training framework designed to significantly reduce memory footprint when training large models. COAT addresses current limitations through two key innovations: (1) Dynamic Range Expansion, which aligns optimizer state distributions more closely with the FP8 representation range, thereby reducing quantization error, and (2) Mixed-Granularity Activation Quantization, which optimizes activation memory using a combination of per-tensor and per-group quantization strategies. Experiments demonstrate that COAT effectively reduces end-to-end training memory footprint by 1.54x compared to BF16 while achieving nearly lossless performance across various tasks, such as Large Language Model pretraining and fine-tuning and Vision Language Model training. COAT also achieves a 1.43x end-to-end training speedup compared to BF16, performing on par with or surpassing TransformerEngine's speedup. COAT enables efficient full-parameter training of large models on fewer GPUs, and facilitates doubling the batch size in distributed training settings, providing a practical solution for scaling large-scale model training. The code is available at https://github.com/NVlabs/COAT.

cs.LG↗

arXiv2024

Detection of low-energy fluxons from engineered long Josephson junctions for efficient computing

Single-Flux Quantum (SFQ) digital logic is typically energy efficient and fast, and logic that uses ballistic and reversible principles provides a new platform to improve efficiency. We are studying long Josephson junctions (long JJs), SFQs within them, and an SFQ detector, all intended for future ballistic logic gate experiments. Specifically, we launch low-energy SFQ into engineered long JJs made from an array of 80 JJs and connecting inductors. The component JJs have critical currents of only 7.5 uA such that the Josephson penetration depth is approximately 2.4 unit cells, and the SFQ's stationary energy in the LJJ is ~47 zJ. The circuit measured consisted of three components: an SFQ launcher, the LJJ, and an SFQ detector that uses JJ critical currents of only 15-20 uA. The circuit was measured in two environments: at 4.2 K in a helium dunk probe and 3.5~K in a cryogen-free refrigerator. According to calculations, the SFQ may traverse the LJJ ballistically, i.e., with a small change in velocity. Data show that SFQ detection events are synchronous with SFQ launch events in both setups. The jitter extracted from the launch and arrival times is predominantly attributed to the noise in the detector. This study shows that we can create and detect low-energy SFQs made from engineered LJJs, and the importance of jitter studies for future ballistic gate measurements.

physics.app-ph↗

arXiv2024

SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers

We introduce Sana, a text-to-image framework that can efficiently generate images up to 4096$\times$4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU. Core designs include: (1) Deep compression autoencoder: unlike traditional AEs, which compress images only 8$\times$, we trained an AE that can compress images 32$\times$, effectively reducing the number of latent tokens. (2) Linear DiT: we replace all vanilla attention in DiT with linear attention, which is more efficient at high resolutions without sacrificing quality. (3) Decoder-only text encoder: we replaced T5 with modern decoder-only small LLM as the text encoder and designed complex human instruction with in-context learning to enhance the image-text alignment. (4) Efficient training and sampling: we propose Flow-DPM-Solver to reduce sampling steps, with efficient caption labeling and selection to accelerate convergence. As a result, Sana-0.6B is very competitive with modern giant diffusion model (e.g. Flux-12B), being 20 times smaller and 100+ times faster in measured throughput. Moreover, Sana-0.6B can be deployed on a 16GB laptop GPU, taking less than 1 second to generate a 1024$\times$1024 resolution image. Sana enables content creation at low cost. Code and model will be publicly released.

cs.CV↗

arXiv2024

HART: Efficient Visual Generation with Hybrid Autoregressive Transformer

We introduce Hybrid Autoregressive Transformer (HART), an autoregressive (AR) visual generation model capable of directly generating 1024x1024 images, rivaling diffusion models in image generation quality. Existing AR models face limitations due to the poor image reconstruction quality of their discrete tokenizers and the prohibitive training costs associated with generating 1024px images. To address these challenges, we present the hybrid tokenizer, which decomposes the continuous latents from the autoencoder into two components: discrete tokens representing the big picture and continuous tokens representing the residual components that cannot be represented by the discrete tokens. The discrete component is modeled by a scalable-resolution discrete AR model, while the continuous component is learned with a lightweight residual diffusion module with only 37M parameters. Compared with the discrete-only VAR tokenizer, our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K, leading to a 31% generation FID improvement from 7.85 to 5.38. HART also outperforms state-of-the-art diffusion models in both FID and CLIP score, with 4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourced at https://github.com/mit-han-lab/hart.

cs.CV↗

arXiv2024

Zak Phase Induced Topological Nonreciprocity

Topological physics provides novel insights for designing functional photonic devices, such as magnetic-free optical diodes, which are important in optical engineering and quantum information processing. Past efforts mostly focus on the topological edge modes in two-dimensional (2D) photonic Chern lattices, which, however, require delicate fabrication and temporal modulation. In particular, the 1D nonreciprocal edge mode needs to be embedded in a 2D lattice, contradicting with the compactness of integrated photonics. To address these challenges, we investigate the optical nonreciprocity of the 1D Su-Schrieffer-Heeger (SSH) superradiance lattices in room-temperature atoms. The probe fields propagating in two opposite directions perceive two different SSH topological phases, which have different absorption spectra due to the interplay between the Zak phase and the thermal motion of atoms, resulting in optical nonreciprocity. Our findings reveal the relationship between 1D topological matter and optical nonreciprocity, simplifying the design of topologically resilient nonreciprocal devices.

physics.optics↗

arXiv2024

Ideal flat and resolved SU(3) Landau levels in three dimensions

Landau levels (LLs) are of great importance for understanding the quantum Hall effect and associated many-body physics. Recently, their three-dimensional (3D) counterparts, i.e., dispersionless 3D LLs with well-defined quantum numbers, have attracted significant attention but have not yet been reported. Here we theoretically propose and experimentally observe 3D LLs with a sharply quantized spectrum in a diamond acoustic lattice, where the eigenstates are characterized by SU(3) quantum numbers. The engineered inhomogeneous hopping strengths not only introduce pseudomagnetic fields that quantize the nodal lines into LLs but also provide three bosonic degrees of freedom, embedding a generic SU(3) symmetry into the LLs. Using a phased array of acoustic sources, we selectively excite distinct eigenstates within the degenerate LL multiplets and visualize their 3D eigenmodes. Importantly, our approach enables the precise reconstruction of SU(3) quantum numbers directly from eigenmode correlations. Our results establish SU(3) LLs as a tractable model in artificial platforms, and pave the way for synthesizing LLs with zero dispersion and countable quantum numbers in arbitrary dimensions.

cond-mat.mes-hall↗

arXiv2024

From photon momentum transfer to acceleration sensing

As a typical application of photon momentum transfer, optical levitation systems are known for their ideal isolation from mechanical dissipation and thermal noise. These characters offer extraordinary potential for acceleration precision sensing and have attracted extensive attention in both fundamental and applied physics. Although considerable improvements of optical levitation accelerometers has been reported, the dynamic testing of the sensing performance remains a crucial challenge before the utilization in practical application scenarios. In this work, we present a dual-beam optical levitation accelerometer and demonstrate the test with dynamic inputs for the first time. An acceleration sensing sensitivity of $0.1μg$ and a measurement range of $ 1g$ are achieved. These advancements solidify the potential of optical levitation accelerometer for deployment in practical domains, including navigation, intelligent driving, and industrial automation, building a bridge between the laboratory systems and real-world applications.

physics.optics↗

arXiv2024

Repairing Schemes for Tamo-Barg Codes

In this paper, the repair problem for erasures beyond locality in locally repairable codes is explored under a practical system setting, where a rack-aware storage system consists of racks, each containing a few parity checks. This is referred to as a rack-aware system with locality. Two repair schemes are devised to reduce the repair bandwidth for Tamo-Barg codes under the rack-aware model by setting each repair set as a rack. Additionally, a cut-set bound for locally repairable codes under the rack-aware model with locality is introduced. Using this bound, the second repair scheme is proven to be optimal. Furthermore, the partial-repair problem is considered for locally repairable codes under the rack-aware model with locality, and both repair schemes and bounds are introduced for this scenario.

cs.IT↗

arXiv2024

MDS Generalized Convertible Code

In this paper, we consider the convertible codes with the maximum distance separable (MDS) property, which can adjust the code rate according to the failure rates of devices. We first extend the notion of convertible codes to allow initial and final codes with different parameters. Then, we investigate the relationship between these parameters and thus establish new lower bounds on the access cost in the merge and split regimes. To gain a deeper understanding of access-optimal MDS convertible codes in the merge regime, we characterize them from the perspective of parity check matrices. Consequently, we present a necessary and sufficient condition for the access-optimal MDS convertible code in the merge regime. Finally, as an application of our characterization, we construct MDS convertible codes in the merge regime with optimal access cost based on the extended generalized Reed-Solomon codes.

cs.IT↗

arXiv2024

DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models

Diffusion models have achieved great success in synthesizing high-quality images. However, generating high-resolution images with diffusion models is still challenging due to the enormous computational costs, resulting in a prohibitive latency for interactive applications. In this paper, we propose DistriFusion to tackle this problem by leveraging parallelism across multiple GPUs. Our method splits the model input into multiple patches and assigns each patch to a GPU. However, naively implementing such an algorithm breaks the interaction between patches and loses fidelity, while incorporating such an interaction will incur tremendous communication overhead. To overcome this dilemma, we observe the high similarity between the input from adjacent diffusion steps and propose displaced patch parallelism, which takes advantage of the sequential nature of the diffusion process by reusing the pre-computed feature maps from the previous timestep to provide context for the current step. Therefore, our method supports asynchronous communication, which can be pipelined by computation. Extensive experiments show that our method can be applied to recent Stable Diffusion XL with no quality degradation and achieve up to a 6.1$\times$ speedup on eight NVIDIA A100s compared to one. Our code is publicly available at https://github.com/mit-han-lab/distrifuser.

cs.CV↗

arXiv2024

Velocity Scanning Tomography for Room-Temperature Quantum Simulation

Quantum simulation offers an analog approach for exploring exotic quantum phenomena using controllable platforms, typically necessitating ultracold temperatures to maintain the quantum coherence. Superradiance lattices (SLs) have been harnessed to simulate coherent topological physics at room temperature, but the thermal motion of atoms remains a notable challenge in accurately measuring the physical quantities. To overcome this obstacle, we invent and validate a velocity scanning tomography technique to discern the responses of atoms with different velocities, allowing cold-atom spectroscopic resolution within room-temperature SLs. By comparing absorption spectra with and without atoms moving at specific velocities, we can derive the Wannier-Stark ladders of the SL across various effective static electric fields, their strengths being proportional to the atomic velocities. We extract the Zak phase of the SL by monitoring the ladder frequency shift as a function of the atomic velocity, effectively demonstrating the topological winding of the energy bands. Our research signifies the feasibility of room-temperature quantum simulation and facilitates their applications in quantum information processing.

quant-ph↗

arXiv2024

A Transformation of Repairing Reed-Solomon Codes from Rack-Aware Storage Model to Homogeneous Storage Model

In this paper, we address the node repair problem of Reed-Solomon (RS) coded distributed storage systems. Specifically, to overcome the challenges of multiple-node failures of RS codes under the rack-aware storage model, we employ good polynomials to guide the placement of the conventional RS codes into racks and then propose a novel repair framework for the resultant rack-aware RS codes, which can transform its repair to that under the homogeneous storage model. As applications of our repair framework, firstly we present the repair scheme of multiple-node failures for some existing constructions, which only have non-trivial solutions for repairing a single-node failure before. Secondly, we deduce several new constructions of rack-aware RS codes supporting the repair of multiple-node failures within a single rack and across multiple racks respectively.

cs.IT↗

arXiv2024

EfficientViT-SAM: Accelerated Segment Anything Model Without Accuracy Loss

We present EfficientViT-SAM, a new family of accelerated segment anything models. We retain SAM's lightweight prompt encoder and mask decoder while replacing the heavy image encoder with EfficientViT. For the training, we begin with the knowledge distillation from the SAM-ViT-H image encoder to EfficientViT. Subsequently, we conduct end-to-end training on the SA-1B dataset. Benefiting from EfficientViT's efficiency and capacity, EfficientViT-SAM delivers 48.9x measured TensorRT speedup on A100 GPU over SAM-ViT-H without sacrificing performance. Our code and pre-trained models are released at https://github.com/mit-han-lab/efficientvit.

cs.CV↗

arXiv2024

Wavelength-tunable high-fidelity entangled photon sources enabled by dual Stark effects

The construction of a large-scale quantum internet requires quantum repeaters containing multiple entangled photon sources with identical wavelengths. Semiconductor quantum dots can generate entangled photon pairs deterministically with high fidelity. However, realizing wavelength-matched quantum-dot entangled photon sources faces two difficulties: the non-uniformity of emission wavelength and exciton fine-structure splitting induced fidelity reduction. Typically, these two factors are not independently tunable, making it challenging to achieve simultaneous improvement. In this work, we demonstrate wavelength-tunable entangled photon sources based on droplet-etched GaAs quantum dots through the combined use of AC and quantum-confined Stark effects. The emission wavelength can be tuned by ~1 meV while preserving an entanglement fidelity f exceeding 0.955(1) in the entire tuning range. Based on this hybrid tuning scheme, we finally demonstrate multiple wavelength-matched entangled photon sources with f>0.919(3), paving a way towards robust and scalable on-demand entangled photon sources for quantum internet and integrated quantum optical circuits.

quant-ph↗

arXiv2024

MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning

Tiny deep learning on microcontroller units (MCUs) is challenging due to the limited memory size. We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs: the first several blocks have an order of magnitude larger memory usage than the rest of the network. To alleviate this issue, we propose a generic patch-by-patch inference scheduling, which operates only on a small spatial region of the feature map and significantly cuts down the peak memory. However, naive implementation brings overlapping patches and computation overhead. We further propose network redistribution to shift the receptive field and FLOPs to the later stage and reduce the computation overhead. Manually redistributing the receptive field is difficult. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2. Patch-based inference effectively reduces the peak memory usage of existing networks by 4-8x. Co-designed with neural networks, MCUNetV2 sets a record ImageNet accuracy on MCU (71.8%), and achieves >90% accuracy on the visual wake words dataset under only 32kB SRAM. MCUNetV2 also unblocks object detection on tiny devices, achieving 16.9% higher mAP on Pascal VOC compared to the state-of-the-art result. Our study largely addressed the memory bottleneck in tinyML and paved the way for various vision applications beyond image classification.

cs.CV↗

arXiv2024

Condition-Aware Neural Network for Controlled Image Generation

We present Condition-Aware Neural Network (CAN), a new method for adding control to image generative models. In parallel to prior conditional control methods, CAN controls the image generation process by dynamically manipulating the weight of the neural network. This is achieved by introducing a condition-aware weight generation module that generates conditional weight for convolution/linear layers based on the input condition. We test CAN on class-conditional image generation on ImageNet and text-to-image generation on COCO. CAN consistently delivers significant improvements for diffusion transformer models, including DiT and UViT. In particular, CAN combined with EfficientViT (CaT) achieves 2.78 FID on ImageNet 512x512, surpassing DiT-XL/2 while requiring 52x fewer MACs per sampling step.

cs.CV↗