arXiv subjects

Tianyu Wang

Publications and source records attributed to Tianyu Wang.

At least 127 records · Page 7Linked to original sources

arXiv2025

DRO: A Python Library for Distributionally Robust Optimization in Machine Learning

We introduce dro, an open-source Python library for distributionally robust optimization (DRO) for regression and classification problems. The library implements 14 DRO formulations and 9 backbone models, enabling 79 distinct DRO methods. Furthermore, dro is compatible with both scikit-learn and PyTorch. Through vectorization and optimization approximation techniques, dro reduces runtime by 10x to over 1000x compared to baseline implementations on large-scale datasets. Comprehensive documentation is available at https://python-dro.org.

cs.LG↗

arXiv2025

Large-scale quantum reservoir computing using a Gaussian Boson Sampler

A Gaussian boson sampler (GBS) is a special-purpose quantum computer that can be practically realized at large scale in optics. Here we report on experiments in which we used a frequency-multiplexed GBS with $>400$ modes as the reservoir in the quantum-machine-learning approach of quantum reservoir computing. We evaluated the accuracy of our GBS-based reservoir computer on a variety of benchmark tasks, including spoken-vowels classification and MNIST handwritten-digit classification. We found that when the reservoir computer was given access to the correlations between measured modes of the GBS, the achieved accuracies were the same or higher than when it was only given access to the mean photon number in each mode -- and in several cases the advantage in accuracy from using the correlations was greater than 20 percentage points. This provides experimental evidence in support of theoretical predictions that access to correlations enhances the power of quantum reservoir computers. We also tested our reservoir computer when operating the reservoir with various sources of classical rather than squeezed (quantum) light and found that using squeezed light consistently resulted in the highest (or tied highest, for simple tasks) accuracies. Our work experimentally establishes that a GBS can be an effective reservoir for quantum reservoir computing and provides a practical platform for experimentally exploring the role of quantumness and correlations in quantum machine learning at very large system sizes.

quant-ph↗

arXiv2025

EdgeOL: Efficient in-situ Online Learning on Edge Devices

Emerging applications, such as robot-assisted eldercare and object recognition, generally employ deep learning neural networks (DNNs) and naturally require: i) handling streaming-in inference requests and ii) adapting to possible deployment scenario changes. Online model fine-tuning is widely adopted to satisfy these needs. However, an inappropriate fine-tuning scheme could involve significant energy consumption, making it challenging to deploy on edge devices. In this paper, we propose EdgeOL, an edge online learning framework that optimizes inference accuracy, fine-tuning execution time, and energy efficiency through both inter-tuning and intra-tuning optimizations. Experimental results show that, on average, EdgeOL reduces overall fine-tuning execution time by 64%, energy consumption by 52%, and improves average inference accuracy by 1.75% over the immediate online learning strategy

cs.LG↗

arXiv2025

Programmable metasurfaces for future photonic artificial intelligence

Photonic neural networks (PNNs), which share the inherent benefits of photonic systems, such as high parallelism and low power consumption, could challenge traditional digital neural networks in terms of energy efficiency, latency, and throughput. However, producing scalable photonic artificial intelligence (AI) solutions remains challenging. To make photonic AI models viable, the scalability problem needs to be solved. Large optical AI models implemented on PNNs are only commercially feasible if the advantages of optical computation outweigh the cost of their input-output overhead. In this Perspective, we discuss how field-programmable metasurface technology may become a key hardware ingredient in achieving scalable photonic AI accelerators and how it can compete with current digital electronic technologies. Programmability or reconfigurability is a pivotal component for PNN hardware, enabling in situ training and accommodating non-stationary use cases that require fine-tuning or transfer learning. Co-integration with electronics, 3D stacking, and large-scale manufacturing of metasurfaces would significantly improve PNN scalability and functionalities. Programmable metasurfaces could address some of the current challenges that PNNs face and enable next-generation photonic AI technology.

physics.optics↗

arXiv2025

3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer

Current 3D Large Multimodal Models (3D LMMs) have shown tremendous potential in 3D-vision-based dialogue and reasoning. However, how to further enhance 3D LMMs to achieve fine-grained scene understanding and facilitate flexible human-agent interaction remains a challenging problem. In this work, we introduce 3D-LLaVA, a simple yet highly powerful 3D LMM designed to act as an intelligent assistant in comprehending, reasoning, and interacting with the 3D world. Unlike existing top-performing methods that rely on complicated pipelines-such as offline multi-view feature extraction or additional task-specific heads-3D-LLaVA adopts a minimalist design with integrated architecture and only takes point clouds as input. At the core of 3D-LLaVA is a new Omni Superpoint Transformer (OST), which integrates three functionalities: (1) a visual feature selector that converts and selects visual tokens, (2) a visual prompt encoder that embeds interactive visual prompts into the visual token space, and (3) a referring mask decoder that produces 3D masks based on text description. This versatile OST is empowered by the hybrid pretraining to obtain perception priors and leveraged as the visual connector that bridges the 3D data to the LLM. After performing unified instruction tuning, our 3D-LLaVA reports impressive results on various benchmarks.

cs.CV↗

arXiv2025

Thermodynamic formalism for non-uniform systems with controlled specification and entropy expansiveness

We study thermodynamic formalism of dynamical systems with non-uniform structure. Precisely, we obtain the uniqueness of equilibrium states for a family of non-uniformly expansive flows by generalizing Climenhaga-Thompson's orbit decomposition criteria. In particular, such family includes entropy expansive flows. Meanwhile, the essential part of the decomposition is allowed to satisfy an even weaker version of specification, namely controlled specification, thus also extends the corresponding results by Pavlov. Two applications of our abstract theorems are explored. Firstly, we introduce a notion of regularity condition called weak Walters condition, and study the uniqueness of measure of maximal entropy for a suspension flow with roof function satisfying such condition. Secondly, we investigate topologically transitive frame flows on rank one manifolds of nonpositive curvature, which is a group extension of nonuniformly hyperbolic flows. Under a bunched curvature condition and running a Gauss-Bonnet type of argument, we show the uniqueness of equilibrium states with respect to certain potentials.

math.DS↗

arXiv2025

Batched Stochastic Bandit for Nondegenerate Functions

This paper studies batched bandit learning problems for nondegenerate functions. We introduce an algorithm that solves the batched bandit problem for nondegenerate functions near-optimally. More specifically, we introduce an algorithm, called Geometric Narrowing (GN), whose regret bound is of order $\widetilde{\mathcal{O}} ( A_{+}^d \sqrt{T} )$. In addition, GN only needs $\mathcal{O} (\log \log T)$ batches to achieve this regret. We also provide lower bound analysis for this problem. More specifically, we prove that over some (compact) doubling metric space of doubling dimension $d$: 1. For any policy $π$, there exists a problem instance on which $π$ admits a regret of order $Ω ( A_-^d \sqrt{T})$; 2. No policy can achieve a regret of order $ A_-^d \sqrt{T} $ over all problem instances, using less than $ Ω( \log \log T ) $ rounds of communications. Our lower bound analysis shows that the GN algorithm achieves near optimal regret with minimal number of batches.

stat.ML↗

arXiv2025

Step-Video-TI2V Technical Report: A State-of-the-Art Text-Driven Image-to-Video Generation Model

We present Step-Video-TI2V, a state-of-the-art text-driven image-to-video generation model with 30B parameters, capable of generating videos up to 102 frames based on both text and image inputs. We build Step-Video-TI2V-Eval as a new benchmark for the text-driven image-to-video task and compare Step-Video-TI2V with open-source and commercial TI2V engines using this dataset. Experimental results demonstrate the state-of-the-art performance of Step-Video-TI2V in the image-to-video generation task. Both Step-Video-TI2V and Step-Video-TI2V-Eval are available at https://github.com/stepfun-ai/Step-Video-TI2V.

cs.CV↗

arXiv2025

ObjectMover: Generative Object Movement with Video Prior

Simple as it seems, moving an object to another location within an image is, in fact, a challenging image-editing task that requires re-harmonizing the lighting, adjusting the pose based on perspective, accurately filling occluded regions, and ensuring coherent synchronization of shadows and reflections while maintaining the object identity. In this paper, we present ObjectMover, a generative model that can perform object movement in highly challenging scenes. Our key insight is that we model this task as a sequence-to-sequence problem and fine-tune a video generation model to leverage its knowledge of consistent object generation across video frames. We show that with this approach, our model is able to adjust to complex real-world scenarios, handling extreme lighting harmonization and object effect movement. As large-scale data for object movement are unavailable, we construct a data generation pipeline using a modern game engine to synthesize high-quality data pairs. We further propose a multi-task learning strategy that enables training on real-world video data to improve the model generalization. Through extensive experiments, we demonstrate that ObjectMover achieves outstanding results and adapts well to real-world scenarios.

cs.GR↗

arXiv2025

HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generation

Co-speech gestures are crucial non-verbal cues that enhance speech clarity and expressiveness in human communication, which have attracted increasing attention in multimodal research. While the existing methods have made strides in gesture accuracy, challenges remain in generating diverse and coherent gestures, as most approaches assume independence among multimodal inputs and lack explicit modeling of their interactions. In this work, we propose a novel multimodal learning method named HOP for co-speech gesture generation that captures the heterogeneous entanglement between gesture motion, audio rhythm, and text semantics, enabling the generation of coordinated gestures. By leveraging spatiotemporal graph modeling, we achieve the alignment of audio and action. Moreover, to enhance modality coherence, we build the audio-text semantic representation based on a reprogramming module, which is beneficial for cross-modality adaptation. Our approach enables the trimodal system to learn each other's features and represent them in the form of topological entanglement. Extensive experiments demonstrate that HOP achieves state-of-the-art performance, offering more natural and expressive co-speech gesture generation. More information, codes, and demos are available here: https://star-uu-wang.github.io/HOP/

cs.CV↗

arXiv2025

Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model

We present Step-Video-T2V, a state-of-the-art text-to-video pre-trained model with 30B parameters and the ability to generate videos up to 204 frames in length. A deep compression Variational Autoencoder, Video-VAE, is designed for video generation tasks, achieving 16x16 spatial and 8x temporal compression ratios, while maintaining exceptional video reconstruction quality. User prompts are encoded using two bilingual text encoders to handle both English and Chinese. A DiT with 3D full attention is trained using Flow Matching and is employed to denoise input noise into latent frames. A video-based DPO approach, Video-DPO, is applied to reduce artifacts and improve the visual quality of the generated videos. We also detail our training strategies and share key observations and insights. Step-Video-T2V's performance is evaluated on a novel video generation benchmark, Step-Video-T2V-Eval, demonstrating its state-of-the-art text-to-video quality when compared with both open-source and commercial engines. Additionally, we discuss the limitations of current diffusion-based model paradigm and outline future directions for video foundation models. We make both Step-Video-T2V and Step-Video-T2V-Eval available at https://github.com/stepfun-ai/Step-Video-T2V. The online version can be accessed from https://yuewen.cn/videos as well. Our goal is to accelerate the innovation of video foundation models and empower video content creators.

cs.CV↗

arXiv2025

Imaging the photochemical dynamics of cyclobutanone with MeV ultrafast electron diffraction

We study the photoinduced chemical dynamics of cyclobutanone upon excitation at 200 nm to the 3s Rydberg state using MeV ultrafast electron diffraction (UED). We observe both the elastic scattering signal, which contains information about the structural dynamics, and the inelastic scattering signal, which encodes information about the electronic state. Our results suggest a sub-picosecond timescale for the photodissociation dynamics, and an excited state lifetime of about 230 femtoseconds. The dissociation is found to be dominated by the C3 channel where cyclopropane and CO are produced. The branching ratio of the C3 channel to the C2 channel where ethene and ketene are produced, is estimated to be approximately 5:3. Our data suggest that the C3 and C2 channels account for approximately 80% of the photoproducts, with the remaining 20% exhibiting ring-opened structures. It is found that the timescale associated with the dissociation process in the C2 channel is shorter compared to that in the C3 channel. Leveraging the enhanced temporal resolution of MeV UED, our results provide a real-time mapping of the nuclear wavepacket dynamics, capturing the complete photochemical dynamics from S2 minimum through the S1/S0 conical intersection, and finally to the dissociation. Our experimental results provide new insights into the Norrish Type I reaction and can be used to benchmark non-adiabatic dynamics simulations.

physics.chem-ph↗

arXiv2025

Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction

Real-time speech interaction, serving as a fundamental interface for human-machine collaboration, holds immense potential. However, current open-source models face limitations such as high costs in voice data collection, weakness in dynamic control, and limited intelligence. To address these challenges, this paper introduces Step-Audio, the first production-ready open-source solution. Key contributions include: 1) a 130B-parameter unified speech-text multi-modal model that achieves unified understanding and generation, with the Step-Audio-Chat version open-sourced; 2) a generative speech data engine that establishes an affordable voice cloning framework and produces the open-sourced lightweight Step-Audio-TTS-3B model through distillation; 3) an instruction-driven fine control system enabling dynamic adjustments across dialects, emotions, singing, and RAP; 4) an enhanced cognitive architecture augmented with tool calling and role-playing abilities to manage complex tasks effectively. Based on our new StepEval-Audio-360 evaluation benchmark, Step-Audio achieves state-of-the-art performance in human evaluations, especially in terms of instruction following. On open-source benchmarks like LLaMA Question, shows 9.3% average performance improvement, demonstrating our commitment to advancing the development of open-source multi-modal language technologies. Our code and models are available at https://github.com/stepfun-ai/Step-Audio.

cs.CL↗

arXiv2025

Investigating social alignment via mirroring in a system of interacting language models

Alignment is a social phenomenon wherein individuals share a common goal or perspective. Mirroring, or mimicking the behaviors and opinions of another individual, is one mechanism by which individuals can become aligned. Large scale investigations of the effect of mirroring on alignment have been limited due to the scalability of traditional experimental designs in sociology. In this paper, we introduce a simple computational framework that enables studying the effect of mirroring behavior on alignment in multi-agent systems. We simulate systems of interacting large language models in this framework and characterize overall system behavior and alignment with quantitative measures of agent dynamics. We find that system behavior is strongly influenced by the range of communication of each agent and that these effects are exacerbated by increased rates of mirroring. We discuss the observed simulated system behavior in the context of known human social dynamics.

cs.MA↗

arXiv2025

Multi-Carrier Thermal Transport in Electronic and Energy Conversion Devices

Nonequilibrium multi-carrier thermal transport is essential for both scientific research and technological applications in electronic, spintronic, and energy conversion devices. This article reviews the fundamentals of phonon, electron, spin, and ion transport driven by temperature gradients in solid-state and soft condensed matters, and the microscopic interactions between energy/charge carriers that can be leveraged for manipulating electrical and thermal transport in energy conversion devices, such as electron-phonon coupling, spin-phonon interaction, and ion-solvent interactions, etc. In coupled electron-phonon transport, we discuss the basics of electron-phonon interactions and their effects on phonon dynamics, thermalization, and nonequilibrium thermal transport. For the phonon-spin interaction, nonequilibrium transport formulation is introduced first, followed by the physics of spin thermoelectric effect and strategies to manipulate them. Contributions to thermal conductivity from magnons as heat carriers are also reviewed. For coupled transport of heat and ions/molecules, we highlight the importance of local molecular configurations that determine the magnitude of the electrochemical gradient, which is the key to improving the efficiency of low-grade heat energy conversion.

cond-mat.mtrl-sci↗

arXiv2025

Quantum-limited stochastic optical neural networks operating at a few quanta per activation

Energy efficiency in computation is ultimately limited by noise, with quantum limits setting the fundamental noise floor. Analog physical neural networks hold promise for improved energy efficiency compared to digital electronic neural networks. However, they are typically operated in a relatively high-power regime so that the signal-to-noise ratio (SNR) is large, and the noise can be treated as a perturbation. We study optical neural networks where all layers except the last are operated in the limit that each neuron can be activated by just a single photon, and as a result the noise on neuron activations is no longer merely perturbative. We show that by using a physics-based probabilistic model of the neuron activations in training, it is possible to perform accurate machine-learning inference in spite of the extremely high shot noise (SNR ~ 1). We experimentally demonstrated MNIST handwritten-digit classification with a test accuracy of 98% using an optical neural network with a hidden layer operating in the single-photon regime; the optical energy used to perform the classification corresponds to just 0.038 photons per multiply-accumulate (MAC) operation. Our physics-aware stochastic training approach might also prove useful with non-optical ultra-low-power hardware.

physics.optics↗

arXiv2025

FastUMI: A Scalable and Hardware-Independent Universal Manipulation Interface with Dataset

Real-world manipulation data involving robotic arms is crucial for developing generalist action policies, yet such data remains scarce since existing data collection methods are hindered by high costs, hardware dependencies, and complex setup requirements. In this work, we introduce FastUMI, a substantial redesign of the Universal Manipulation Interface (UMI) system that addresses these challenges by enabling rapid deployment, simplifying hardware-software integration, and delivering robust performance in real-world data acquisition. Compared with UMI, FastUMI has several advantages: 1) It adopts a decoupled hardware design and incorporates extensive mechanical modifications, removing dependencies on specialized robotic components while preserving consistent observation perspectives. 2) It also refines the algorithmic pipeline by replacing complex Visual-Inertial Odometry (VIO) implementations with an off-the-shelf tracking module, significantly reducing deployment complexity while maintaining accuracy. 3) FastUMI includes an ecosystem for data collection, verification, and integration with both established and newly developed imitation learning algorithms, accelerating policy learning advancement. Additionally, we have open-sourced a high-quality dataset of over 10,000 real-world demonstration trajectories spanning 22 everyday tasks, forming one of the most diverse UMI-like datasets to date. Experimental results confirm that FastUMI facilitates rapid deployment, reduces operational costs and labor demands, and maintains robust performance across diverse manipulation scenarios, thereby advancing scalable data-driven robotic learning.

cs.RO↗

arXiv2025

An Adaptable Budget Planner for Enhancing Budget-Constrained Auto-Bidding in Online Advertising

In online advertising, advertisers commonly utilize auto-bidding services to bid for impression opportunities. A typical objective of the auto-bidder is to optimize the advertiser's cumulative value of winning impressions within specified budget constraints. However, such a problem is challenging due to the complex bidding environment faced by diverse advertisers. To address this challenge, we introduce ABPlanner, a few-shot adaptable budget planner designed to improve budget-constrained auto-bidding. ABPlanner is based on a hierarchical bidding framework that decomposes the bidding process into shorter, manageable stages. Within this framework, ABPlanner allocates the budget across all stages, allowing a low-level auto-bidder to bids based on the budget allocation plan. The adaptability of ABPlanner is achieved through a sequential decision-making approach, inspired by in-context reinforcement learning. For each advertiser, ABPlanner adjusts the budget allocation plan episode by episode, using data from previous episodes as prompt for current decisions. This enables ABPlanner to quickly adapt to different advertisers with few-shot data, providing a sample-efficient solution. Extensive simulation experiments and real-world A/B testing validate the effectiveness of ABPlanner, demonstrating its capability to enhance the cumulative value achieved by auto-bidders.

cs.GT↗