arXiv subjects

Chong Zhang

Publications and source records attributed to Chong Zhang.

At least 109 records · Page 6Linked to original sources

arXiv2024

OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning

We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets.

cs.RO↗

arXiv2024

Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis

Recent language model-based text-to-speech (TTS) frameworks demonstrate scalability and in-context learning capabilities. However, they suffer from robustness issues due to the accumulation of errors in speech unit predictions during autoregressive language modeling. In this paper, we propose a phonetic enhanced language modeling method to improve the performance of TTS models. We leverage self-supervised representations that are phonetically rich as the training target for the autoregressive language model. Subsequently, a non-autoregressive model is employed to predict discrete acoustic codecs that contain fine-grained acoustic details. The TTS model focuses solely on linguistic modeling during autoregressive training, thereby reducing the error propagation that occurs in non-autoregressive training. Both objective and subjective evaluations validate the effectiveness of our proposed method.

eess.AS↗

arXiv2024

Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study

Scaling text-to-speech (TTS) with autoregressive language model (LM) to large-scale datasets by quantizing waveform into discrete speech tokens is making great progress to capture the diversity and expressiveness in human speech, but the speech reconstruction quality from discrete speech token is far from satisfaction depending on the compressed speech token compression ratio. Generative diffusion models trained with score-matching loss and continuous normalized flow trained with flow-matching loss have become prominent in generation of images as well as speech. LM based TTS systems usually quantize speech into discrete tokens and generate these tokens autoregressively, and finally use a diffusion model to up sample coarse-grained speech tokens into fine-grained codec features or mel-spectrograms before reconstructing into waveforms with vocoder, which has a high latency and is not realistic for real time speech applications. In this paper, we systematically investigate varied diffusion models for up sampling stage, which is the main bottleneck for streaming synthesis of LM and diffusion-based architecture, we present the model architecture, objective and subjective metrics to show quality and efficiency improvement.

eess.AS↗

arXiv2024

Rethinking Robustness Assessment: Adversarial Attacks on Learning-based Quadrupedal Locomotion Controllers

Legged locomotion has recently achieved remarkable success with the progress of machine learning techniques, especially deep reinforcement learning (RL). Controllers employing neural networks have demonstrated empirical and qualitative robustness against real-world uncertainties, including sensor noise and external perturbations. However, formally investigating the vulnerabilities of these locomotion controllers remains a challenge. This difficulty arises from the requirement to pinpoint vulnerabilities across a long-tailed distribution within a high-dimensional, temporally sequential space. As a first step towards quantitative verification, we propose a computational method that leverages sequential adversarial attacks to identify weaknesses in learned locomotion controllers. Our research demonstrates that, even state-of-the-art robust controllers can fail significantly under well-designed, low-magnitude adversarial sequence. Through experiments in simulation and on the real robot, we validate our approach's effectiveness, and we illustrate how the results it generates can be used to robustify the original policy and offer valuable insights into the safety of these black-box policies. Project page: https://fanshi14.github.io/me/rss24.html

cs.RO↗

arXiv2024

Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion

Legged robots navigating cluttered environments must be jointly agile for efficient task execution and safe to avoid collisions with obstacles or humans. Existing studies either develop conservative controllers (< 1.0 m/s) to ensure safety, or focus on agility without considering potentially fatal collisions. This paper introduces Agile But Safe (ABS), a learning-based control framework that enables agile and collision-free locomotion for quadrupedal robots. ABS involves an agile policy to execute agile motor skills amidst obstacles and a recovery policy to prevent failures, collaboratively achieving high-speed and collision-free navigation. The policy switch in ABS is governed by a learned control-theoretic reach-avoid value network, which also guides the recovery policy as an objective function, thereby safeguarding the robot in a closed loop. The training process involves the learning of the agile policy, the reach-avoid value network, the recovery policy, and an exteroception representation network, all in simulation. These trained modules can be directly deployed in the real world with onboard sensing and computation, leading to high-speed and collision-free navigation in confined indoor and outdoor spaces with both static and dynamic obstacles.

cs.RO↗

arXiv2024

Better Sampling, towards Better End-to-end Small Object Detection

While deep learning-based general object detection has made significant strides in recent years, the effectiveness and efficiency of small object detection remain unsatisfactory. This is primarily attributed not only to the limited characteristics of such small targets but also to the high density and mutual overlap among these targets. The existing transformer-based small object detectors do not leverage the gap between accuracy and inference speed. To address challenges, we propose methods enhancing sampling within an end-to-end framework. Sample Points Refinement (SPR) constrains localization and attention, preserving meaningful interactions in the region of interest and filtering out misleading information. Scale-aligned Target (ST) integrates scale information into target confidence, improving classification for small object detection. A task-decoupled Sample Reweighting (SR) mechanism guides attention toward challenging positive examples, utilizing a weight generator module to assess the difficulty and adjust classification loss based on decoder layer outcomes. Comprehensive experiments across various benchmarks reveal that our proposed detector excels in detecting small objects. Our model demonstrates a significant enhancement, achieving a 2.9\% increase in average precision (AP) over the state-of-the-art (SOTA) on the VisDrone dataset and a 1.7\% improvement on the SODA-D dataset.

cs.CV↗

arXiv2024

All-Optical Manipulation of Band Gap Dynamics via Electron-Phonon Coupling

The electron-phonon coupling (EPC) is a ubiquitous interaction in condensed systems and plays a vital role in shaping the electronic properties of materials. Yet, achieving coherent manipulation of electron-phonon coupling has posed a considerable challenge. Here, employing time-resolved high-harmonic generation (tr-HHG) spectroscopy, we demonstrate the coherent manipulation of bandgap dynamics in a BaF2 crystal by precisely controlling the EPC using ultrashort light pulses. The tr-HHG spectrum perturbed by a triply degenerate phonon mode T2g, exhibits simultaneously a remarkable two-dimensional (2D) sensitivity, namely intensity domain in addition to the previously reported energy domain. The dynamic compression and enhancement of the harmonics in the intensity domain showed a π/2 phase shift compared to the manifestation of shifts of the harmonics in the energy domain, an astounding example of a physical phenomenon being observed simultaneously in two different perspectives. To complement our experimental observations, we employed a quantum model that incorporates the EPC, successfully reproducing the results. In addition, we demonstrated complete control over the EPC strength and initial phase of the coherent phonon oscillations by varying the incident electric field polarization over crystal orientation. Our findings lay a foundation for future investigations aiming to harness and exploit the remarkable potential of EPC in solid-state systems.

cond-mat.mtrl-sci↗

arXiv2024

SPGM: Prioritizing Local Features for enhanced speech separation performance

Dual-path is a popular architecture for speech separation models (e.g. Sepformer) which splits long sequences into overlapping chunks for its intra- and inter-blocks that separately model intra-chunk local features and inter-chunk global relationships. However, it has been found that inter-blocks, which comprise half a dual-path model's parameters, contribute minimally to performance. Thus, we propose the Single-Path Global Modulation (SPGM) block to replace inter-blocks. SPGM is named after its structure consisting of a parameter-free global pooling module followed by a modulation module comprising only 2% of the model's total parameters. The SPGM block allows all transformer layers in the model to be dedicated to local feature modelling, making the overall model single-path. SPGM achieves 22.1 dB SI-SDRi on WSJ0-2Mix and 20.4 dB SI-SDRi on Libri2Mix, exceeding the performance of Sepformer by 0.5 dB and 0.3 dB respectively and matches the performance of recent SOTA models with up to 8 times fewer parameters. Model and weights are available at huggingface.co/yipjiaqi/spgm

eess.AS↗

arXiv2024

Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation

We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. To create a large-scale retargeted motion dataset of human movements for humanoid robots, we propose a scalable "sim-to-data" process to filter and pick feasible motions using a privileged motion imitator. Afterwards, we train a robust real-time humanoid motion imitator in simulation using these refined motions and transfer it to the real humanoid robot in a zero-shot manner. We successfully achieve teleoperation of dynamic whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based real-time whole-body humanoid teleoperation.

cs.RO↗

arXiv2024

Light-induced giant enhancement of nonreciprocal transport at KTaO3-based interfaces

Nonlinear transport is a unique functionality of noncentrosymmetric systems, which reflects profound physics, such as spin-orbit interaction, superconductivity and band geometry. However, it remains highly challenging to enhance the nonreciprocal transport for promising rectification devices. Here, we observe a light-induced giant enhancement of nonreciprocal transport at the superconducting and epitaxial CaZrO3/KTaO3 (111) interfaces. The nonreciprocal transport coefficient undergoes a giant increase with three orders of magnitude up to 105 A-1T-1. Furthermore, a strong Rashba spin-orbit coupling effective field of 14.7 T is achieved with abundant high-mobility photocarriers under ultraviolet illumination, which accounts for the giant enhancement of nonreciprocal transport coefficient. Our first-principles calculations further disclose the stronger Rashba spin-orbit coupling strength and the longer relaxation time in the photocarrier excitation process, bridging the light-property quantitative relationship. Our work provides an alternative pathway to boost nonreciprocal transport in noncentrosymmetric systems and facilitates the promising applications in opto-rectification devices and spin-orbitronic devices.

cond-mat.mes-hall↗

arXiv2024

Towards Training A Chinese Large Language Model for Anesthesiology

Medical large language models (LLMs) have gained popularity recently due to their significant practical utility. However, most existing research focuses on general medicine, and there is a need for in-depth study of LLMs in specific fields like anesthesiology. To fill the gap, we introduce Hypnos, a Chinese Anesthesia model built upon existing LLMs, e.g., Llama. Hypnos' contributions have three aspects: 1) The data, such as utilizing Self-Instruct, acquired from current LLMs likely includes inaccuracies. Hypnos implements a cross-filtering strategy to improve the data quality. This strategy involves using one LLM to assess the quality of the generated data from another LLM and filtering out the data with low quality. 2) Hypnos employs a general-to-specific training strategy that starts by fine-tuning LLMs using the general medicine data and subsequently improving the fine-tuned LLMs using data specifically from Anesthesiology. The general medical data supplement the medical expertise in Anesthesiology and enhance the effectiveness of Hypnos' generation. 3) We introduce a standardized benchmark for evaluating medical LLM in Anesthesiology. Our benchmark includes both publicly available instances from the Internet and privately obtained cases from the Hospital. Hypnos outperforms other medical LLMs in anesthesiology in metrics, GPT-4, and human evaluation on the benchmark dataset.

cs.CL↗

arXiv2024

Learning Highly Dynamic Behaviors for Quadrupedal Robots

Learning highly dynamic behaviors for robots has been a longstanding challenge. Traditional approaches have demonstrated robust locomotion, but the exhibited behaviors lack diversity and agility. They employ approximate models, which lead to compromises in performance. Data-driven approaches have been shown to reproduce agile behaviors of animals, but typically have not been able to learn highly dynamic behaviors. In this paper, we propose a learning-based approach to enable robots to learn highly dynamic behaviors from animal motion data. The learned controller is deployed on a quadrupedal robot and the results show that the controller is able to reproduce highly dynamic behaviors including sprinting, jumping and sharp turning. Various behaviors can be activated through human interaction using a stick with markers attached to it. Based on the motion pattern of the stick, the robot exhibits walking, running, sitting and jumping, much like the way humans interact with a pet.

cs.RO↗

arXiv2024

Loss Masking Is Not Needed in Decoder-only Transformer for Discrete-token-based ASR

Recently, unified speech-text models, such as SpeechGPT, VioLA, and AudioPaLM, have achieved remarkable performance on various speech tasks. These models discretize speech signals into tokens (speech discretization) and use a shared vocabulary for both text and speech tokens. Then they train a single decoder-only Transformer on a mixture of speech tasks. However, these models rely on the Loss Masking strategy for the ASR task, which ignores the dependency among speech tokens. In this paper, we propose to model speech tokens in an autoregressive way, similar to text. We find that applying the conventional cross-entropy loss on input speech tokens does not consistently improve the ASR performance over the Loss Masking approach. To address this issue, we propose a novel approach denoted Smoothed Label Distillation (SLD), which applies a KL divergence loss with smoothed labels on speech tokens. Our experiments show that SLD effectively models speech tokens and outperforms Loss Masking for decoder-only Transformers in ASR tasks with different speech discretization methods. The source code can be found here: https://github.com/alibaba-damo-academy/SpokenNLP/tree/main/sld

cs.CL↗

arXiv2024

Multi-Model Wireless Federated Learning with Downlink Beamforming

This paper studies the design of wireless federated learning (FL) for simultaneously training multiple machine learning models. We consider round robin device-model assignment and downlink beamforming for concurrent multiple model updates. After formulating the joint downlink-uplink transmission process, we derive the per-model global update expression over communication rounds, capturing the effect of beamforming and noisy reception. To maximize the multi-model training convergence rate, we derive an upper bound on the optimality gap of the global model update and use it to formulate a multi-group multicast beamforming problem. We show that this problem can be converted to minimizing the sum of inverse received signal-to-interference-plus-noise ratios, which can be solved efficiently by projected gradient descent. Simulation shows that our proposed multi-model FL solution outperforms other alternatives, including conventional single-model sequential training and multi-model zero-forcing beamforming.

cs.IT↗

arXiv2023

Bio-Image Informatics Index BIII: A unique database of image analysis tools and workflows for and by the bioimaging community

Bio image analysis has recently become one keystone of biological research but biologists tend to get lost in a plethora of available software and the way to adjust available tools to their own image analysis problem. We present BIII, BioImage Informatic Index (www.biii.eu), the result of the first large community effort to bridge the communities of algorithm and software developers, bioimage analysts and biologists, under the form of a web-based knowledge database crowdsourced by these communities. Software tools (> 1300), image databases for benchmarking (>20) and training materials (>70) for bio image analysis are referenced and curated following standards constructed by the community and then reaching a broader audience. Software tools are organized as full protocol of analysis (workflow), specific brick (component) to construct a workflow, or software platform or library (collection). They are described using Edam Bio Imaging, which is iteratively defined using this website. All entries are exposed following FAIR principles and accessible for other usage.

q-bio.QM↗

arXiv2023

The super approximation property of $\mathrm{SL}_2(\mathbb{Z}/q\mathbb{Z}) \times \mathrm{SL}_2(\mathbb{Z}/q\mathbb{Z}) \times \mathrm{SL}_2(\mathbb{Z}/q\mathbb{Z})$

Take $S \subset \mathrm{SL}_2(\mathbb{Z}) \times \mathrm{SL}_2(\mathbb{Z})\times \mathrm{SL}_2(\mathbb{Z})$ be finite symmetric and assume $S$ generates a group $G$ which is Zariski-dense in $\mathrm{SL}_2 \times \mathrm{SL}_2\times \mathrm{SL}_2(\mathbb{Z})$. This paper proves that the Cayley graphs $$ \{\mathcal{C} a y(G(\bmod q), S(\bmod q))\}_{q \in \mathbb{Z}_{+}} $$ form a family of expanders.

math.GR↗

arXiv2023

Image Blending Algorithm with Automatic Mask Generation

In recent years, image blending has gained popularity for its ability to create visually stunning content. However, the current image blending algorithms mainly have the following problems: manually creating image blending masks requires a lot of manpower and material resources; image blending algorithms cannot effectively solve the problems of brightness distortion and low resolution. To this end, we propose a new image blending method with automatic mask generation: it combines semantic object detection and segmentation with mask generation to achieve deep blended images based on our proposed new saturation loss and two-stage iteration of the PAN algorithm to fix brightness distortion and low-resolution issues. Results on publicly available datasets show that our method outperforms other classical image blending algorithms on various performance metrics, including PSNR and SSIM.

cs.CV↗

arXiv2023

Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings

Prior studies diagnose the anisotropy problem in sentence representations from pre-trained language models, e.g., BERT, without fine-tuning. Our analysis reveals that the sentence embeddings from BERT suffer from a bias towards uninformative words, limiting the performance in semantic textual similarity (STS) tasks. To address this bias, we propose a simple and efficient unsupervised approach, Diagonal Attention Pooling (Ditto), which weights words with model-based importance estimations and computes the weighted average of word representations from pre-trained models as sentence embeddings. Ditto can be easily applied to any pre-trained language model as a postprocessing operation. Compared to prior sentence embedding approaches, Ditto does not add parameters nor requires any learning. Empirical evaluations demonstrate that our proposed Ditto can alleviate the anisotropy problem and improve various pre-trained models on STS tasks.

cs.CL↗