arXiv subjects

Haotian Yang

Publications and source records attributed to Haotian Yang.

At least 37 records · Page 2Linked to original sources

arXiv2025

Imbalance in Balance: Online Concept Balancing in Generation Models

In visual generation tasks, the responses and combinations of complex concepts often lack stability and are error-prone, which remains an under-explored area. In this paper, we attempt to explore the causal factors for poor concept responses through elaborately designed experiments. We also design a concept-wise equalization loss function (IMBA loss) to address this issue. Our proposed method is online, eliminating the need for offline dataset processing, and requires minimal code changes. In our newly proposed complex concept benchmark Inert-CompBench and two other public test sets, our method significantly enhances the concept response capability of baseline models and yields highly competitive results with only a few codes released at https://github.com/KwaiVGI/IMBA-Loss.

cs.CV↗

arXiv2025

TGT: Text-Grounded Trajectories for Locally Controlled Video Generation

Text-to-video generation has advanced rapidly in visual fidelity, whereas standard methods still have limited ability to control the subject composition of generated scenes. Prior work shows that adding localized text control signals, such as bounding boxes or segmentation masks, can help. However, these methods struggle in complex scenarios and degrade in multi-object settings, offering limited precision and lacking a clear correspondence between individual trajectories and visual entities as the number of controllable objects increases. We introduce Text-Grounded Trajectories (TGT), a framework that conditions video generation on trajectories paired with localized text descriptions. We propose Location-Aware Cross-Attention (LACA) to integrate these signals and adopt a dual-CFG scheme to separately modulate local and global text guidance. In addition, we develop a data processing pipeline that produces trajectories with localized descriptions of tracked entities, and we annotate two million high quality video clips to train TGT. Together, these components enable TGT to use point trajectories as intuitive motion handles, pairing each trajectory with text to control both appearance and motion. Extensive experiments show that TGT achieves higher visual quality, more accurate text alignment, and improved motion controllability compared with prior approaches. Website: https://textgroundedtraj.github.io.

cs.CV↗

arXiv2025

Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs

Recent advances in diffusion-based text-to-image (T2I) models have led to remarkable success in generating high-quality images from textual prompts. However, ensuring accurate alignment between the text and the generated image remains a significant challenge for state-of-the-art diffusion models. To address this, existing studies employ reinforcement learning with human feedback (RLHF) to align T2I outputs with human preferences. These methods, however, either rely directly on paired image preference data or require a learned reward function, both of which depend heavily on costly, high-quality human annotations and thus face scalability limitations. In this work, we introduce Text Preference Optimization (TPO), a framework that enables "free-lunch" alignment of T2I models, achieving alignment without the need for paired image preference data. TPO works by training the model to prefer matched prompts over mismatched prompts, which are constructed by perturbing original captions using a large language model. Our framework is general and compatible with existing preference-based algorithms. We extend both DPO and KTO to our setting, resulting in TDPO and TKTO. Quantitative and qualitative evaluations across multiple benchmarks show that our methods consistently outperform their original counterparts, delivering better human preference scores and improved text-to-image alignment. Our Open-source code is available at https://github.com/DSL-Lab/T2I-Free-Lunch-Alignment.

cs.CV↗

arXiv2025

A Hybrid Force-Position Strategy for Shape Control of Deformable Linear Objects With Graph Attention Networks

Manipulating deformable linear objects (DLOs) such as wires and cables is crucial in various applications like electronics assembly and medical surgeries. However, it faces challenges due to DLOs' infinite degrees of freedom, complex nonlinear dynamics, and the underactuated nature of the system. To address these issues, this paper proposes a hybrid force-position strategy for DLO shape control. The framework, combining both force and position representations of DLO, integrates state trajectory planning in the force space and Model Predictive Control (MPC) in the position space. We present a dynamics model with an explicit action encoder, a property extractor and a graph processor based on Graph Attention Networks. The model is used in the MPC to enhance prediction accuracy. Results from both simulations and real-world experiments demonstrate the effectiveness of our approach in achieving efficient and stable shape control of DLOs. Codes and videos are available at https://sites.google.com/view/dlom.

cs.RO↗

arXiv2025

VMoBA: Mixture-of-Block Attention for Video Diffusion Models

The quadratic complexity of full attention mechanisms poses a significant bottleneck for Video Diffusion Models (VDMs) aiming to generate long-duration, high-resolution videos. While various sparse attention methods have been proposed, many are designed as training-free inference accelerators or do not optimally capture the unique spatio-temporal characteristics inherent in video data when trained natively. This paper introduces Video Mixture of Block Attention (VMoBA), a novel sparse attention mechanism specifically adapted for VDMs. Motivated by an in-depth analysis of attention patterns within pre-trained video transformers, which revealed strong spatio-temporal locality, varying query importance, and head-specific concentration levels, VMoBA enhances the original MoBA framework with three key modifications: (1) a layer-wise recurrent block partition scheme (1D-2D-3D) to dynamically adapt to diverse spatio-temporal attention patterns and improve efficiency; (2) global block selection to prioritize the most salient query-key block interactions across an entire attention head; and (3) threshold-based block selection to dynamically determine the number of attended blocks based on their cumulative similarity. Extensive experiments demonstrate that VMoBA significantly accelerates the training of VDMs on longer sequences, achieving 2.92x FLOPs and 1.48x latency speedup, while attaining comparable or even superior generation quality to full attention. Furthermore, VMoBA exhibits competitive performance in training-free inference, offering 2.40x FLOPs and 1.35x latency speedup for high-res video generation.

cs.CV↗

arXiv2025

Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content

With the continuous progress of visual generation technologies, the scale of video datasets has grown exponentially. The quality of these datasets plays a pivotal role in the performance of video generation models. We assert that temporal splitting, detailed captions, and video quality filtering are three crucial determinants of dataset quality. However, existing datasets exhibit various limitations in these areas. To address these challenges, we introduce Koala-36M, a large-scale, high-quality video dataset featuring accurate temporal splitting, detailed captions, and superior video quality. The essence of our approach lies in improving the consistency between fine-grained conditions and video content. Specifically, we employ a linear classifier on probability distributions to enhance the accuracy of transition detection, ensuring better temporal consistency. We then provide structured captions for the splitted videos, with an average length of 200 words, to improve text-video alignment. Additionally, we develop a Video Training Suitability Score (VTSS) that integrates multiple sub-metrics, allowing us to filter high-quality videos from the original corpus. Finally, we incorporate several metrics into the training process of the generation model, further refining the fine-grained conditions. Our experiments demonstrate the effectiveness of our data processing pipeline and the quality of the proposed Koala-36M dataset. Our dataset and code have been released at https://koala36m.github.io/.

cs.CV↗

arXiv2025

An Empirical Study of the Impact of Federated Learning on Machine Learning Model Accuracy

Federated Learning (FL) enables distributed ML model training on private user data at the global scale. Despite the potential of FL demonstrated in many domains, an in-depth view of its impact on model accuracy remains unclear. In this paper, we investigate, systematically, how this learning paradigm can affect the accuracy of state-of-the-art ML models for a variety of ML tasks. We present an empirical study that involves various data types: text, image, audio, and video, and FL configuration knobs: data distribution, FL scale, client sampling, and local and global computations. Our experiments are conducted in a unified FL framework to achieve high fidelity, with substantial human efforts and resource investments. Based on the results, we perform a quantitative analysis of the impact of FL, and highlight challenging scenarios where applying FL degrades the accuracy of the model drastically and identify cases where the impact is negligible. The detailed and extensive findings can benefit practical deployments and future development of FL.

cs.LG↗

arXiv2025

DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers

Diffusion models have demonstrated remarkable success in various image generation tasks, but their performance is often limited by the uniform processing of inputs across varying conditions and noise levels. To address this limitation, we propose a novel approach that leverages the inherent heterogeneity of the diffusion process. Our method, DiffMoE, introduces a batch-level global token pool that enables experts to access global token distributions during training, promoting specialized expert behavior. To unleash the full potential of the diffusion process, DiffMoE incorporates a capacity predictor that dynamically allocates computational resources based on noise levels and sample complexity. Through comprehensive evaluation, DiffMoE achieves state-of-the-art performance among diffusion models on ImageNet benchmark, substantially outperforming both dense architectures with 3x activated parameters and existing MoE approaches while maintaining 1x activated parameters. The effectiveness of our approach extends beyond class-conditional generation to more challenging tasks such as text-to-image generation, demonstrating its broad applicability across different diffusion model applications. Project Page: https://shiml20.github.io/DiffMoE/

cs.CV↗

arXiv2025

Separating hypergraph Turán densities

Determining the Turán densities of hypergraphs is a notoriously difficult problem at the core of combinatorics. Although Turán posed this problem in 1941, $π(K_{\ell}^{(k)})$ remains unknown for all $\ell>k\geq 3$. Prior to this work, it was not even known whether $π(K_{\ell}^{(k)})<π(K_{\ell+1}^{(k)})$ holds for general $\ell$ and $k$, and the best-known bounds on $π(K_{\ell}^{(k)})$ are far from implying anything close to this. We prove that $π(K_{\ell}^{(k)})<π(K_{\ell+1}^{(k)})$, for all $\ell>k\geq 3$, and provide a general criterion to distinguish the Turán densities of two hypergraphs. As a corollary, we obtain that $π(K_{k+1}^{(k)})<π(K_{k+2}^{(k)-})$, for all $k\geq 3$. For $k=3$, this was previously proved by Markström, answering a question by Erdős.

math.CO↗

arXiv2024

Topological cliques in sparse expanders

In the paper, we focus on embedding clique immersions and subdivisions within sparse expanders, and we derive the following main results: (1) For any $0< η< 1/2$, there exists $K>0$ such that for sufficiently large $n$, every $(n,d,λ)$-graph $G$ contains a $K_{(1-5η)d}$-immersion when $d\geq Kλ$. (2) For any $\varepsilon>0$ and $0<η<1/2$, the following holds for sufficiently large $n$. Every $(n,d,λ)$-graph $G$ with $2048λ/η^2 0$ such that the following holds for sufficiently large $d$. If $G$ is an $n$-vertex graph with average degree $d(G)\geq d$, then $G$ contains a $K_{c d}^{(\ell)}$-immersion for some $\ell\in \mathbb{N}$. In 2018, Dvo{ř}{á}k and Yepremyan asked whether every graph $G$ with $δ(G)\geq t$ contains a $K_t$-immersion. Our first result shows that it is asymptotically true for $(n,d,λ)$-graphs when $λ=o(d)$. In addition, our second result extends a result of Dragani{ć}, Krivelevich and Nenadov on balanced subdivisions. The last result generalises a result of DeVos, Dvo{ř}{á}k, Fox, McDonald, Mohar, Scheide on $1$-immersions of large cliques in dense graphs.

math.CO↗

arXiv2024

VideoTetris: Towards Compositional Text-to-Video Generation

Diffusion models have demonstrated great success in text-to-video (T2V) generation. However, existing methods may face challenges when handling complex (long) video generation scenarios that involve multiple objects or dynamic changes in object numbers. To address these limitations, we propose VideoTetris, a novel framework that enables compositional T2V generation. Specifically, we propose spatio-temporal compositional diffusion to precisely follow complex textual semantics by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, we propose an enhanced video data preprocessing to enhance the training data regarding motion dynamics and prompt understanding, equipped with a new reference frame attention mechanism to improve the consistency of auto-regressive video generation. Extensive experiments demonstrate that our VideoTetris achieves impressive qualitative and quantitative results in compositional T2V generation. Code is available at: https://github.com/YangLing0818/VideoTetris

cs.CV↗

arXiv2024

On $3$-graphs with vanishing codegree Turán density

For a $k$-uniform hypergraph (or simply $k$-graph) $F$, the codegree Turán density $π_{\mathrm{co}}(F)$ is the supremum over all $α$ such that there exist arbitrarily large $n$-vertex $F$-free $k$-graphs $H$ in which every $(k-1)$-subset of $V(H)$ is contained in at least $αn$ edges. Recently, it was proved that for every $3$-graph $F$, $π_{\mathrm{co}}(F)=0$ implies $π_{\therefore}(F)=0$, where $π_{\therefore}(F)$ is the uniform Turán density of $F$ and is defined as the supremum over all $d$ such that there are infinitely many $F$-free $k$-graphs $H$ satisfying that any induced linear-size subhypergraph of $H$ has edge density at least $d$. In this paper, we introduce a layered structure for $3$-graphs which allows us to obtain the reverse implication: every layered $3$-graph $F$ with $π_{\therefore}(F)=0$ satisfies $π_{\mathrm{co}}(F)=0$. Along the way, we answer in the negative a question of Falgas-Ravry, Pikhurko, Vaughan and Volec [J. London Math. Soc., 2023] about whether $π_{\therefore}(F)\leqπ_{\mathrm{co}}(F)$ always holds. In particular, we construct counterexamples $F$ with positive but arbitrarily small $π_{\mathrm{co}}(F)$ while having $π_{\therefore}(F)\ge 4/27$.

math.CO↗

arXiv2024

VRMM: A Volumetric Relightable Morphable Head Model

In this paper, we introduce the Volumetric Relightable Morphable Model (VRMM), a novel volumetric and parametric facial prior for 3D face modeling. While recent volumetric prior models offer improvements over traditional methods like 3D Morphable Models (3DMMs), they face challenges in model learning and personalized reconstructions. Our VRMM overcomes these by employing a novel training framework that efficiently disentangles and encodes latent spaces of identity, expression, and lighting into low-dimensional representations. This framework, designed with self-supervised learning, significantly reduces the constraints for training data, making it more feasible in practice. The learned VRMM offers relighting capabilities and encompasses a comprehensive range of expressions. We demonstrate the versatility and effectiveness of VRMM through various applications like avatar generation, facial reconstruction, and animation. Additionally, we address the common issue of overfitting in generative volumetric models with a novel prior-preserving personalization framework based on VRMM. Such an approach enables high-quality 3D face reconstruction from even a single portrait input. Our experiments showcase the potential of VRMM to significantly enhance the field of 3D face modeling.

cs.CV↗

arXiv2024

Text-Driven Diverse Facial Texture Generation via Progressive Latent-Space Refinement

Automatic 3D facial texture generation has gained significant interest recently. Existing approaches may not support the traditional physically based rendering pipeline or rely on 3D data captured by Light Stage. Our key contribution is a progressive latent space refinement approach that can bootstrap from 3D Morphable Models (3DMMs)-based texture maps generated from facial images to generate high-quality and diverse PBR textures, including albedo, normal, and roughness. It starts with enhancing Generative Adversarial Networks (GANs) for text-guided and diverse texture generation. To this end, we design a self-supervised paradigm to overcome the reliance on ground truth 3D textures and train the generative model with only entangled texture maps. Besides, we foster mutual enhancement between GANs and Score Distillation Sampling (SDS). SDS boosts GANs with more generative modes, while GANs promote more efficient optimization of SDS. Furthermore, we introduce an edge-aware SDS for multi-view consistent facial structure. Experiments demonstrate that our method outperforms existing 3D texture generation methods regarding photo-realistic quality, diversity, and efficiency.

cs.CV↗

arXiv2023

Vanishing codegree Turán density implies vanishing uniform Turán density

For a $k$-uniform hypergraph (or simply $k$-graph) $F$, the codegree Turán density $π_{\mathrm{co}}(F)$ is the infimum over all $α$ such that any $n$-vertex $k$-graph $H$ with every $(k-1)$-subset of $V(H)$ contained in at least $αn$ edges has a copy of $F$. The uniform Turán density $π_{\therefore}(F)$ is the supremum over all $d$ such that there are infinitely many $F$-free $k$-graphs $H$ satisfying that any linear-size subhypergraph of $H$ has edge density at least $d$. Falgas-Ravry, Pikhurko, Vaughan and Volec [J. London Math. Soc., 2023] asked whether for every $3$-graph $F$, $π_{\therefore}(F)\leqπ_{\mathrm{co}}(F)$. We provide a positive answer to this question provided that $π_{\mathrm{co}}(F)=0$. Our proof relies on a random geometric construction and a new formulation of the characterization of $3$-graphs with vanishing uniform Turán density due to Reiher, R{ö}dl and Schacht [J. London Math. Soc., 2018]. Along the way, we answer a question of Falgas-Ravry, Pikhurko, Vaughan and Volec about subhypergraphs with linear minimum codegree in uniformly dense hypergraphs in the negative.

math.CO↗

arXiv2023

FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction

In this paper, we present a large-scale detailed 3D face dataset, FaceScape, and the corresponding benchmark to evaluate single-view facial 3D reconstruction. By training on FaceScape data, a novel algorithm is proposed to predict elaborate riggable 3D face models from a single image input. FaceScape dataset releases $16,940$ textured 3D faces, captured from $847$ subjects and each with $20$ specific expressions. The 3D models contain the pore-level facial geometry that is also processed to be topologically uniform. These fine 3D facial models can be represented as a 3D morphable model for coarse shapes and displacement maps for detailed geometry. Taking advantage of the large-scale and high-accuracy dataset, a novel algorithm is further proposed to learn the expression-specific dynamic details using a deep neural network. The learned relationship serves as the foundation of our 3D face prediction system from a single image input. Different from most previous methods, our predicted 3D models are riggable with highly detailed geometry under different expressions. We also use FaceScape data to generate the in-the-wild and in-the-lab benchmark to evaluate recent methods of single-view face reconstruction. The accuracy is reported and analyzed on the dimensions of camera pose and focal length, which provides a faithful and comprehensive evaluation and reveals new challenges. The unprecedented dataset, benchmark, and code have been released at https://github.com/zhuhao-nju/facescape.

cs.CV↗

arXiv2023

Towards Practical Capture of High-Fidelity Relightable Avatars

In this paper, we propose a novel framework, Tracking-free Relightable Avatar (TRAvatar), for capturing and reconstructing high-fidelity 3D avatars. Compared to previous methods, TRAvatar works in a more practical and efficient setting. Specifically, TRAvatar is trained with dynamic image sequences captured in a Light Stage under varying lighting conditions, enabling realistic relighting and real-time animation for avatars in diverse scenes. Additionally, TRAvatar allows for tracking-free avatar capture and obviates the need for accurate surface tracking under varying illumination conditions. Our contributions are two-fold: First, we propose a novel network architecture that explicitly builds on and ensures the satisfaction of the linear nature of lighting. Trained on simple group light captures, TRAvatar can predict the appearance in real-time with a single forward pass, achieving high-quality relighting effects under illuminations of arbitrary environment maps. Second, we jointly optimize the facial geometry and relightable appearance from scratch based on image sequences, where the tracking is implicitly learned. This tracking-free approach brings robustness for establishing temporal correspondences between frames under different lighting conditions. Extensive qualitative and quantitative experiments demonstrate that our framework achieves superior performance for photorealistic avatar animation and relighting.

cs.CV↗

arXiv2023

Enhanced mechanical performance and bioactivity in strontium/copper co-substituted diopside scaffolds

Effective scaffolds for bone tissue-engineering are those that combine adequate mechanical and chemical performance with osseointegrative, angiogenetic and anti-bacterial modes of bioactivity. To address these requirements via a combined approach, we additively manufactured square strut scaffolds by robocasting precipitation-derived strontium/copper co-substituted diopside. Microstructure, mechanical performance, bioactivity, biocompatibility, and antibacterial activity were examined. The results show that the presence of strontium and copper in the diopside lattice reduces the grain size and increases the density of the ceramics. The compressive strength, hardness, and fracture toughness of the diopside showed improvement, attributed to a finer microstructure and improved sintering. Scaffolds had excellent compressive strength with a high porosity (68 to 72 %), which is attributed to the structure of the stacked square struts. All materials showed good in vitro bioactivity and favorable proliferation of osteogenic sarcoma cells, while strontium and copper co-doped materials exhibited the strongest anti-Escherichia coli activity. We show that across multiple indicators this system offers pathways towards high-performance bone substitutes.

cond-mat.mtrl-sci↗