arXiv subjects

Luyu Wang

Publications and source records attributed to Luyu Wang.

At least 19 recordsLinked to original sources

arXiv2026

Bridging Data, Reasoning, and Alignment: A Unified Framework for Context-Aware Instruction-Following TTS

The ISCSLP 2026 CoT-TTS Challenge requires TTS systems to generate Chain-of-Thought (CoT) reasoning from dialogue history before synthesizing contextually appropriate speech. While the official baseline establishes a unified architecture, it remains constrained by limited contextual comprehension, weak instruction fidelity, and suboptimal audio quality. We present a systematic optimization pipeline to address these limitations. First, we develop a data process framework that cleans raw data via FullSubNet denoising, Qwen3-ASR re-transcription, and Qwen3.5-35B-A3B-based history-CoT consistency analysis, while distilling 545K high-fidelity instruction samples using Qwen3-TTS and Seed-VC under strict quality filtration. Second, we propose a Context-Aware Direct Preference Optimization (CA-DPO) method. By employing a cascaded filtering strategy, ASR prescreening, LLM tournament ranking, and speaker similarity verification, we obtain high-confidence preference pairs that significantly enhance holistic ``Context$\rightarrow$CoT$\rightarrow$Speech'' consistency during DPO training. Third, we establish an evaluation method featuring a 500-sample test set and an LLM-as-Judge framework to independently assess reasoning and execution fidelity. Experiments demonstrate that our system significantly outperforms the baseline across all objective and subjective metrics, validating our data governance and alignment strategies.

eess.AS↗

arXiv2026

Modified uni-travelling-carrier photodiodes with 206 GHz bandwidth and 0.81 A/W external responsivity

The accelerating demand for wireless communication necessitates wideband, energy-efficient photonic sub-terahertz (sub-THz) sources to enable ultra-fast data transfer. However, as critical components for THz photonic mixing, photodiodes (PDs) face a fundamental trade-off between quantum efficiency and bandwidth, presenting a major obstacle to achieving high-speed performance with high optoelectronic conversion efficiency. Here, we overcome this challenge by demonstrating an InP-based, waveguide-integrated modified uni-traveling carrier photodiode (MUTC-PD) with bandwidth exceeding 200 GHz and a bandwidth-efficiency product (BEP) surpassing 130 GHz. Incorporating a spot-size converter (SSC) together with optimized electric-field distribution, balanced carrier transport, and minimized parasitic capacitance, the device achieves a 3-dB bandwidth of 206 GHz and an external responsivity of 0.81 A/W, setting a new BEP benchmark. Packaged with WR-5.1 waveguide output, it delivers radio-frequency (RF) power exceeding -5 dBm across the 127-185 GHz frequency range. As a proof of concept, we achieved a wireless transmission of 54 meters with a single-line rate of up to 120 Gbps, leveraging photonics-aided technology without requiring a low-noise amplifier (LNA). This work establishes a pathway to significantly enhance optical power budgets and reduce energy consumption, presenting a transformative step toward high-bandwidth, high-efficiency sub-THz communication systems and next-generation wireless networks.

physics.app-ph↗

arXiv2025

Scalable Optical Links for Controlling Bosonic Quantum Processors

Superconducting quantum computing has the potential to revolutionize computational capabilities. However, scaling up large quantum processors is limited by the cumbersome and heat-conductive electronic cables that connect room-temperature control electronics to quantum processors, leading to significant signal attenuation. Optical fibers provide a promising solution, but their use has been restricted to controlling simple two-level quantum systems over short distances. Here, we demonstrate optical control of a bosonic quantum processor, achieving universal operations on the joint Hilbert space of a transmon qubit and a storage cavity. Using an array of cryogenic fiber-integrated uni-traveling-carrier photodiodes, we prepare Fock states containing up to ten photons. Additionally, remote control of bosonic modes over a transmission distance of 15 km has been achieved, with fidelities exceeding 95%. The combination of high-dimensional quantum control, multi-channel operation, and long-distance transmission addresses the key requirements for scaling superconducting quantum computers and enables architectures for distributed quantum data centers.

quant-ph↗

arXiv2025

SIMA 2: A Generalist Embodied Agent for Virtual Worlds

We introduce SIMA 2, a generalist embodied agent that understands and acts in a wide variety of 3D virtual worlds. Built upon a Gemini foundation model, SIMA 2 represents a significant step toward active, goal-directed interaction within an embodied environment. Unlike prior work (e.g., SIMA 1) limited to simple language commands, SIMA 2 acts as an interactive partner, capable of reasoning about high-level goals, conversing with the user, and handling complex instructions given through language and images. Across a diverse portfolio of games, SIMA 2 substantially closes the gap with human performance and demonstrates robust generalization to previously unseen environments, all while retaining the base model's core reasoning capabilities. Furthermore, we demonstrate a capacity for open-ended self-improvement: by leveraging Gemini to generate tasks and provide rewards, SIMA 2 can autonomously learn new skills from scratch in a new environment. This work validates a path toward creating versatile and continuously learning agents for both virtual and, eventually, physical worlds.

cs.AI↗

arXiv2024

Vision-Language Models as a Source of Rewards

Building generalist agents that can accomplish many goals in rich open-ended environments is one of the research frontiers for reinforcement learning. A key limiting factor for building generalist agents with RL has been the need for a large number of reward functions for achieving different goals. We investigate the feasibility of using off-the-shelf vision-language models, or VLMs, as sources of rewards for reinforcement learning agents. We show how rewards for visual achievement of a variety of language goals can be derived from the CLIP family of models, and used to train RL agents that can achieve a variety of language goals. We showcase this approach in two distinct visual domains and present a scaling trend showing how larger VLMs lead to more accurate rewards for visual goal achievement, which in turn produces more capable RL agents.

cs.LG↗

arXiv2024

Ultra-fast Waveguide MUTC Photodiodes over 220 GHz

We present InP-based evanescently-coupled waveguide modified uni-traveling carrier photodiodes (MUTC-PDs) exhibiting a breakthrough in bandwidth. The optimization of carrier transport and optical coupling is achieved through a detailed discussion on the design of the cliff layer and waveguide layer. Addressing the parasitic capacitance challenge, we introduce benzocyclobutene (BCB) beneath the PD electrodes, effectively overcoming the bandwidth bottleneck associated with the RC time constant. Devices with sizes of 2 * 7 um2 and 2 * 10 um2 achieve 3-dB bandwidths over 220 GHz, along with external responsivities of 0.161 A/W and 0.237 A/W, respectively. Notably, the RF output power reaches a peak of -1.69 dBm at 215 GHz for 2 * 15 um2 PDs.

physics.app-ph↗

arXiv2023

Zorro: the masked multimodal transformer

Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring very little fusion engineering. The resulting representations are however fully entangled throughout the network, which may not always be desirable: in learning, contrastive audio-visual self-supervised learning requires independent audio and visual features to operate, otherwise learning collapses; in inference, evaluation of audio-visual models should be possible on benchmarks having just audio or just video. In this paper, we introduce Zorro, a technique that uses masks to control how inputs from each modality are routed inside Transformers, keeping some parts of the representation modality-pure. We apply this technique to three popular transformer-based architectures (ViT, Swin and HiP) and show that with contrastive pre-training Zorro achieves state-of-the-art results on most relevant benchmarks for multimodal tasks (AudioSet and VGGSound). Furthermore, the resulting models are able to perform unimodal inference on both video and audio benchmarks such as Kinetics-400 or ESC-50.

cs.CV↗

arXiv2022

In-context Reinforcement Learning with Algorithm Distillation

We propose Algorithm Distillation (AD), a method for distilling reinforcement learning (RL) algorithms into neural networks by modeling their training histories with a causal sequence model. Algorithm Distillation treats learning to reinforcement learn as an across-episode sequential prediction problem. A dataset of learning histories is generated by a source RL algorithm, and then a causal transformer is trained by autoregressively predicting actions given their preceding learning histories as context. Unlike sequential policy prediction architectures that distill post-learning or expert sequences, AD is able to improve its policy entirely in-context without updating its network parameters. We demonstrate that AD can reinforcement learn in-context in a variety of environments with sparse rewards, combinatorial task structure, and pixel-based observations, and find that AD learns a more data-efficient RL algorithm than the one that generated the source data.

cs.LG↗

arXiv2022

Towards Learning Universal Audio Representations

The ability to learn universal audio representations that can solve diverse speech, music, and environment tasks can spur many applications that require general sound content understanding. In this work, we introduce a holistic audio representation evaluation suite (HARES) spanning 12 downstream tasks across audio domains and provide a thorough empirical study of recent sound representation learning systems on that benchmark. We discover that previous sound event classification or speech models do not generalize outside of their domains. We observe that more robust audio representations can be learned with the SimCLR objective; however, the model's transferability depends heavily on the model architecture. We find the Slowfast architecture is good at learning rich representations required by different domains, but its performance is affected by the normalization scheme. Based on these findings, we propose a novel normalizer-free Slowfast NFNet and achieve state-of-the-art performance across all domains.

cs.SD↗

arXiv2021

A stabilized mixed finite element scheme for frictional contact mechanics and shear failure analyses in deformable media with crossing fractures

Simulation of frictional contact and shear failure of fractures in fractured media is of paramount important in computational mechanics. In this work, a preconditioned mixed-finite element (FE) scheme with Lagrange multipliers is proposed in the framework of constrained variational principle, which has the capability to handle frictional contact and slip of the multiple crossing fractures. The slippage, opening and contact traction on fractures are calculated by the resulted saddle-point algebraic system. A novel treatment is devised to guarantee physical solutions at the intersected position of crossing fractures. A preconditioning technique is introduced to re-scale the resulting saddle-point algebraic system, to preserve the robustness of the system. An iteration strategy, namely monolithic-updated contact algorithm, is then designed to update the two primary unknowns (displacement and Lagrange multiplier) in one algebraic block. Then, a series of numerical tests is conducted to study the frictional contact and shear failure of single- and multi-crossing fractures. Benchmark study is performed to verify the presented mixed-FE scheme. Two tests with crossing fractures are studied, in which the slippage and opening can be calculated. The effects of crossing fractures on the deformation field are observed in the simulation, in which the variation of slippage, opening and stress intensity factor are analyzed under different loading conditions.

math.NA↗

arXiv2021

Broaden Your Views for Self-Supervised Video Learning

Most successful self-supervised learning methods are trained to align the representations of two independent views from the data. State-of-the-art methods in video are inspired by image techniques, where these two views are similarly extracted by cropping and augmenting the resulting crop. However, these methods miss a crucial element in the video domain: time. We introduce BraVe, a self-supervised learning framework for video. In BraVe, one of the views has access to a narrow temporal window of the video while the other view has a broad access to the video content. Our models learn to generalise from the narrow view to the general content of the video. Furthermore, BraVe processes the views with different backbones, enabling the use of alternative augmentations or modalities into the broad view such as optical flow, randomly convolved RGB frames, audio or their combinations. We demonstrate that BraVe achieves state-of-the-art results in self-supervised representation learning on standard video and audio classification benchmarks including UCF101, HMDB51, Kinetics, ESC-50 and AudioSet.

cs.CV↗

arXiv2021

WikiGraphs: A Wikipedia Text - Knowledge Graph Paired Dataset

We present a new dataset of Wikipedia articles each paired with a knowledge graph, to facilitate the research in conditional text generation, graph generation and graph representation learning. Existing graph-text paired datasets typically contain small graphs and short text (1 or few sentences), thus limiting the capabilities of the models that can be learned on the data. Our new dataset WikiGraphs is collected by pairing each Wikipedia article from the established WikiText-103 benchmark (Merity et al., 2016) with a subgraph from the Freebase knowledge graph (Bollacker et al., 2008). This makes it easy to benchmark against other state-of-the-art text generative models that are capable of generating long paragraphs of coherent text. Both the graphs and the text data are of significantly larger scale compared to prior graph-text paired datasets. We present baseline graph neural network and transformer model results on our dataset for 3 tasks: graph -> text generation, graph -> text retrieval and text -> graph retrieval. We show that better conditioning on the graph provides gains in generation and retrieval quality but there is still large room for improvement.

cs.CL↗

arXiv2021

Multimodal Self-Supervised Learning of General Audio Representations

We present a multimodal framework to learn general audio representations from videos. Existing contrastive audio representation learning methods mainly focus on using the audio modality alone during training. In this work, we show that additional information contained in video can be utilized to greatly improve the learned features. First, we demonstrate that our contrastive framework does not require high resolution images to learn good audio features. This allows us to scale up the training batch size, while keeping the computational load incurred by the additional video modality to a reasonable level. Second, we use augmentations that mix together different samples. We show that this is effective to make the proxy task harder, which leads to substantial performance improvements when increasing the batch size. As a result, our audio model achieves a state-of-the-art of 42.4 mAP on the AudioSet classification downstream task, closing the gap between supervised and self-supervised methods trained on the same dataset. Moreover, we show that our method is advantageous on a broad range of non-semantic audio tasks, including speaker identification, keyword spotting, language identification, and music instrument classification.

cs.SD↗

arXiv2021

Multi-Format Contrastive Learning of Audio Representations

Recent advances suggest the advantage of multi-modal training in comparison with single-modal methods. In contrast to this view, in our work we find that similar gain can be obtained from training with different formats of a single modality. In particular, we investigate the use of the contrastive learning framework to learn audio representations by maximizing the agreement between the raw audio and its spectral representation. We find a significant gain using this multi-format strategy against the single-format counterparts. Moreover, on the downstream AudioSet and ESC-50 classification task, our audio-only approach achieves new state-of-the-art results with a mean average precision of 0.376 and an accuracy of 90.5%, respectively.

cs.SD↗

arXiv2020

Structures and Properties of $β$-Titanium Doping Trace Transition Metal Elements: a Density Functional Theory Study

We systematically calculate the structure, formation enthalpy, formation free energy, elastic constants and electronic structure of Ti$_{0.98}$X$_{0.02}$ system by density functional theory (DFT) simulations to explore the effect of transition metal X (X=Ag, Cd, Co, Cr, Cu, Fe, Mn, Mo, Nb, Ni, Pd, Rh, Ru, Tc, and Zn) on the stability mechanism of $β$-titanium. Based on our calculations, the results of formation enthalpy and free energy show that adding trace X is beneficial to the thermodynamic stability of $β$-titanium. This behavior is well explained by the density of state (DOS). However, the tetragonal shear moduli of Ti$_{0.98}$X$_{0.02}$ systems are negative, indicating that $β$-titanium doping with a low concentration of X is still elastically unstable at 0 K. Therefore, we theoretically explain that $β$-titanium doping with trace transition metal X is unstable in the ground state.

cond-mat.mtrl-sci↗

arXiv2020

Learning Robust and Multilingual Speech Representations

Unsupervised speech representation learning has shown remarkable success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance. However, most research has been focused on evaluating the representations in terms of their ability to improve the performance of speech recognition systems on read English (e.g. Wall Street Journal and LibriSpeech). This evaluation methodology overlooks two important desiderata that speech representations should have: robustness to domain shifts and transferability to other languages. In this paper we learn representations from up to 8000 hours of diverse and noisy speech data and evaluate the representations by looking at their robustness to domain shifts and their ability to improve recognition performance in many languages. We find that our representations confer significant robustness advantages to the resulting recognition systems: we see significant improvements in out-of-domain transfer relative to baseline feature sets and the features likewise provide improvements in 25 phonetically diverse languages including tonal languages and low-resource languages.

cs.CL↗

arXiv2019

On the Sensitivity of Adversarial Robustness to Input Data Distributions

Neural networks are vulnerable to small adversarial perturbations. Existing literature largely focused on understanding and mitigating the vulnerability of learned models. In this paper, we demonstrate an intriguing phenomenon about the most popular robust training method in the literature, adversarial training: Adversarial robustness, unlike clean accuracy, is sensitive to the input data distribution. Even a semantics-preserving transformations on the input data distribution can cause a significantly different robustness for the adversarial trained model that is both trained and evaluated on the new distribution. Our discovery of such sensitivity on data distribution is based on a study which disentangles the behaviors of clean accuracy and robust accuracy of the Bayes classifier. Empirical investigations further confirm our finding. We construct semantically-identical variants for MNIST and CIFAR10 respectively, and show that standardly trained models achieve comparable clean accuracies on them, but adversarially trained models achieve significantly different robustness accuracies. This counter-intuitive phenomenon indicates that input data distribution alone can affect the adversarial robustness of trained neural networks, not necessarily the tasks themselves. Lastly, we discuss the practical implications on evaluating adversarial robustness, and make initial attempts to understand this complex phenomenon.

cs.LG↗

arXiv2019

advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch

advertorch is a toolbox for adversarial robustness research. It contains various implementations for attacks, defenses and robust training methods. advertorch is built on PyTorch (Paszke et al., 2017), and leverages the advantages of the dynamic computational graph to provide concise and efficient reference implementations. The code is licensed under the LGPL license and is open sourced at https://github.com/BorealisAI/advertorch .

cs.LG↗