arXiv subjects

Zhiyuan Zhao

Publications and source records attributed to Zhiyuan Zhao.

At least 91 records · Page 5Linked to original sources

arXiv2023

MLLM-DataEngine: An Iterative Refinement Approach for MLLM

Despite the great advance of Multimodal Large Language Models (MLLMs) in both instruction dataset building and benchmarking, the independence of training and evaluation makes current MLLMs hard to further improve their capability under the guidance of evaluation results with a relatively low human cost. In this paper, we propose MLLM-DataEngine, a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop iteration, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results, then generate a proper incremental dataset for the next training iteration and enhance the model capability iteratively. Compared with previous data collection methods which are separate from the benchmarking, the data generated by MLLM-DataEngine shows better targeting, quality, and correctness. For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data within each incremental dataset based on the benchmarking results. For quality, we resort to GPT-4 to generate high-quality data with each given data type. For correctness, prompt design is critical for the data generation results. Rather than previous hand-crafted prompt, we propose an Interactive Prompt Optimization strategy, which optimizes the prompt with the multi-round interaction between human and GPT, and improve the correctness of generated data greatly. Through extensive experiments, we find our MLLM-DataEngine could boost the MLLM capability in a targeted and automatic manner, with only a few human participation. We hope it could be a general solution for the following MLLMs building. The MLLM-DataEngine has been open-sourced and is now available at https://github.com/opendatalab/MLLM-DataEngine.

cs.LG↗

arXiv2023

Point convolutional neural network algorithm for Ising model ground state research based on spring vibration

The ground state search of the Ising model can be used to solve many combinatorial optimization problems. Under the current computer architecture, an Ising ground state search algorithm suitable for hardware computing is necessary for solving practical problems. Inspired by the potential energy conversion of springs, we propose a point convolutional neural network algorithm for ground state search based on spring vibration model, called Spring-Ising Algorithm. Spring-Ising Algorithm regards the spin as a moving mass point connected to a spring and establish the equation of motion for all spins. Spring-Ising Algorithm can be mapped on the GPU or AI chips through the basic structure of the neural network for fast and efficient parallel computing. The algorithm has very productive results for solving the Ising model and has been test in the recognized test benchmark K2000. The algorithm introduces the concept of dynamic equilibrium to achieve a more detailed local search by dynamically adjusting the weight of the Ising model in the spring oscillation model. Finally, there is the simple hardware test speed evaluation. Spring-Ising Algorithm can provide the possibility to calculate the Ising model on a chip which focuses on accelerating neural network calculations.

physics.comp-ph↗

arXiv2023

Filler Word Detection with Hard Category Mining and Inter-Category Focal Loss

Filler words like ``um" or ``uh" are common in spontaneous speech. It is desirable to automatically detect and remove them in recordings, as they affect the fluency, confidence, and professionalism of speech. Previous studies and our preliminary experiments reveal that the biggest challenge in filler word detection is that fillers can be easily confused with other hard categories like ``a" or ``I". In this paper, we propose a novel filler word detection method that effectively addresses this challenge by adding auxiliary categories dynamically and applying an additional inter-category focal loss. The auxiliary categories force the model to explicitly model the confusing words by mining hard categories. In addition, inter-category focal loss adaptively adjusts the penalty weight between ``filler" and ``non-filler" categories to deal with other confusing words left in the ``non-filler" category. Our system achieves the best results, with a huge improvement compared to other methods on the PodcastFillers dataset.

eess.AS↗

arXiv2022

Sub-nanotesla Sensitivity at the Nanoscale with a Single Spin

High-sensitivity detection of microscopic magnetic field is essential in many fields. Good sensitivity and high spatial resolution are mutually contradictory in measurement, which is quantified by the energy resolution limit (ERL). Here we report that a sensitivity of 0.5 ${\bf{nT/\sqrt{Hz}}}$ at the nanoscale is achieved experimentally by using nitrogen-vacancy defects in diamond with depths of tens of nanometers. The achieved sensitivity is substantially enhanced by integrating with multiple quantum techniques, including real-time-feedback initialization, dynamical decoupling with shaped pulses, repetitive readout via quantum logic. Our magnetic sensors will shed new light on searching new physics beyond the standard model, investigating microscopic magnetic phenomena in condensed matters, and detection of life activities at the sub-cellular scale.

quant-ph↗

arXiv2022

99.92%-Fidelity CNOT Gates in Solids by Filtering Time-dependent and Quantum Noises

Inevitable interactions with the reservoir largely degrade the performance of non-local gates, which hinders practical quantum computation from coming into existence. Here we experimentally demonstrate a 99.920(7)\%-fidelity controlled-NOT gate by suppressing the complicated noise in a solid-state spin system at room temperature. We found that the fidelity limited at 99\% in previous works results from only considering static noise, and thus, in this work, time-dependent noise and quantum noise are also included. All noises are dynamically corrected by an exquisitely designed shaped pulse, giving the resulting error below $10^{-4}$. The residual gate error is mainly originated from the longitudinal relaxation and the waveform distortion that can both be further reduced technically. Our noise-resistant method is universal, and will benefit other solid-state spin systems.

quant-ph↗

arXiv2022

Temperature-dependent behaviors of single spin defects in solids determined with Hz-level precision

Revealing the properties of single spin defects in solids is essential for quantum applications based on solid-state systems. However, it is intractable to investigate the temperature-dependent properties of single defects, due to the low precision for single-defect measurements in contrast to defect ensembles. Here we report that the temperature dependence of the Hamiltonian parameters for single negatively charged nitrogen-vacancy (NV$^{-}$) centers in diamond is precisely measured, and the results find a reasonable agreement with first-principles calculations. Particularly, the hyperfine interactions with randomly distributed $^{13}$C nuclear spins are clearly observed to vary with temperature, and the relevant coefficients are measured with Hz-level precision. The temperature-dependent behaviors are attributed to both thermal expansion and lattice vibrations by first-principles calculations. Our results pave the way for taking nuclear spins as more stable thermometers at nanoscale. The methods developed here for high-precision measurements and first-principles calculations can be further extended to other solid-state spin defects.

quant-ph↗

arXiv2022

TridentSE: Guiding Speech Enhancement with 32 Global Tokens

In this paper, we present TridentSE, a novel architecture for speech enhancement, which is capable of efficiently capturing both global information and local details. TridentSE maintains T-F bin level representation to capture details, and uses a small number of global tokens to process the global information. Information is propagated between the local and the global representations through cross attention modules. To capture both inter- and intra-frame information, the global tokens are divided into two groups to process along the time and the frequency axis respectively. A metric discriminator is further employed to guide our model to achieve higher perceptual quality. Even with significantly lower computational cost, TridentSE outperforms a variety of previous speech enhancement methods, achieving a PESQ of 3.47 on VoiceBank+DEMAND dataset and a PESQ of 3.44 on DNS no-reverb test set. Visualization shows that the global tokens learn diverse and interpretable global patterns.

eess.AS↗

arXiv2022

Exploring Effective Knowledge Transfer for Few-shot Object Detection

Recently, few-shot object detection~(FSOD) has received much attention from the community, and many methods are proposed to address this problem from a knowledge transfer perspective. Though promising results have been achieved, these methods fail to achieve shot-stable:~methods that excel in low-shot regimes are likely to struggle in high-shot regimes, and vice versa. We believe this is because the primary challenge of FSOD changes when the number of shots varies. In the low-shot regime, the primary challenge is the lack of inner-class variation. In the high-shot regime, as the variance approaches the real one, the main hindrance to the performance comes from misalignment between learned and true distributions. However, these two distinct issues remain unsolved in most existing FSOD methods. In this paper, we propose to overcome these challenges by exploiting rich knowledge the model has learned and effectively transferring them to the novel classes. For the low-shot regime, we propose a distribution calibration method to deal with the lack of inner-class variation problem. Meanwhile, a shift compensation method is proposed to compensate for possible distribution shift during fine-tuning. For the high-shot regime, we propose to use the knowledge learned from ImageNet as guidance for the feature learning in the fine-tuning stage, which will implicitly align the distributions of the novel classes. Although targeted toward different regimes, these two strategies can work together to further improve the FSOD performance. Experiments on both the VOC and COCO benchmarks show that our proposed method can significantly outperform the baseline method and produce competitive results in both low-shot settings (shot<5) and high-shot settings (shot>=5). Code is available at https://github.com/JulioZhao97/EffTrans_Fsdet.git.

cs.CV↗

arXiv2022

Observation of Floquet topological phases with large Chern numbers

One of the most intriguing advantage of Floquet engineering is to generate new phases with large topological invariants. In this work, we experimentally simulate a periodically quenched generalized Haldane model on an NV center in diamond, and observe its Floquet Chern insulator phases with Chern numbers $C=1,2,4$ by imaging the static and dynamic spin textures in momentum space. Our work reveals the power of Floquet driving in transforming system's topology and generating large Chern number phases. It further establishes a unique experimental scheme to detect Floquet topological phases in two and higher spatial dimensions.

quant-ph↗

arXiv2022

PetLock:A Genderless and Standard Interface for the Future On-orbit Construction

Modular design is the foundation of on orbit construction technology of large space facilities in the future.Standard interface is the key technology of modular design of the future space robotic systems and space facilities.This paper presents the designed and tested of PetLock,a standard and genderless interface which can transfer mechanical loads,power and data between the future modular space robotic manipulator and spacecraft.PetLock adopts a completely genderless design,including connection face,locking mechanism,data and power interface.The connection surface provides a large translation and rotation misalignment tolerance,due to its 120-degree symmetrical and 3D shape design.The locking mechanism features the three locking pins retraction structure design,which is simple and reliable.POGO pin connectors in the center of the interface provides the power and data transfer capabilities.Due to the advantages of high locking force,large tolerance,high reliability and low cost,PetLock has the very big application potential in future on orbit construction missions.

cs.RO↗

arXiv2022

A Combined Inverse Kinematics Algorithm Using FABRIK with Optimization

Forward and backward reaching inverse kinematics (FABRIK) is a heuristic inverse kinematics solver that is gradually applied to manipulators with the advantages of fast convergence and generating more realistic configurations. However, under the high error constraint, FABRIK exhibits unstable convergence behavior, which is unsatisfactory for the real-time motion planning of manipulators. In this paper, a novel inverse kinematics algorithm that combines FABRIK and the sequential quadratic programming (SQP) algorithm is presented, in which the joint angles deduced by FABRIK will be taken as the initial seed of the SQP algorithm to avoid getting stuck in local minima. The combined algorithm is evaluated with experiments, in which our algorithm can achieve higher success rates and faster solution times than FABRIK under the high error constraint. Furthermore, the combined algorithm can generate continuous trajectories for the UR5 and KUKA LBR IIWA 14 R820 manipulators in path tracking with no pose error and permitted position error of the end-effector.

cs.RO↗

arXiv2022

An Anchor-Free Detector for Continuous Speech Keyword Spotting

Continuous Speech Keyword Spotting (CSKWS) is a task to detect predefined keywords in a continuous speech. In this paper, we regard CSKWS as a one-dimensional object detection task and propose a novel anchor-free detector, named AF-KWS, to solve the problem. AF-KWS directly regresses the center locations and lengths of the keywords through a single-stage deep neural network. In particular, AF-KWS is tailored for this speech task as we introduce an auxiliary unknown class to exclude other words from non-speech or silent background. We have built two benchmark datasets named LibriTop-20 and continuous meeting analysis keywords (CMAK) dataset for CSKWS. Evaluations on these two datasets show that our proposed AF-KWS outperforms reference schemes by a large margin, and therefore provides a decent baseline for future research.

eess.AS↗

arXiv2022

RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion

This paper proposes a new "decompose-and-edit" paradigm for the text-based speech insertion task that facilitates arbitrary-length speech insertion and even full sentence generation. In the proposed paradigm, global and local factors in speech are explicitly decomposed and separately manipulated to achieve high speaker similarity and continuous prosody. Specifically, we proposed to represent the global factors by multiple tokens, which are extracted by cross-attention operation and then injected back by link-attention operation. Due to the rich representation of global factors, we manage to achieve high speaker similarity in a zero-shot manner. In addition, we introduce a prosody smoothing task to make the local prosody factor context-aware and therefore achieve satisfactory prosody continuity. We further achieve high voice quality with an adversarial training stage. In the subjective test, our method achieves state-of-the-art performance in both naturalness and similarity. Audio samples can be found at https://ydcustc.github.io/retrieverTTS-demo/.

eess.AS↗

arXiv2022

Odd Viscosity in Chiral Passive Suspensions

Prior studies have revealed that nonzero odd viscosity is an essential property for chiral active fluids. Here we report that such an odd viscosity also exists in suspensions of non-active or non-externally-driven but chirally-shaped particles. Computational simulations are carried out for monolayers of dense ratchets in simple shear and planar extensional flows. The contact between two ratchets can be either frictionless or infinitely-frictional, depending on their teeth and sliding directions at the contact point. Our results show that the ratchet suspension has the intermediate shear/extensional viscosity as compared with the suspensions of smooth and gear-like particles. Meanwhile, the ratchet suspensions show nonzero even and odd components of the first normal stress coefficient, which indicates the mixed feature of conventional complex fluids and chiral viscous fluids.

cond-mat.soft↗

arXiv2022

Emergent Stripes of Active Rotors in Shear Flows

The shear-induced self-organization of active rotors into stripy aggregates is studied by carrying out computational simulations. The rotors, modeled by monolayers of frictional spheres, develop to stripy microstructures only when they counterrotate with respect to the vorticity of the imposed shear flow. The average width of the stripes is demonstrated to be linearly dependent on the relative intensity of active torque to the shear rate. By giving insight into three collective particle behaviors, i.e., shear-induced diffusion, rotation-induced rearrangement, and edge flows, we explain the mechanisms of formation of the particle stripes. Additionally, the rheological result shows the dependence of shear and rotational viscosities on the active torque direction and the oddness of the normal stress response. By exhibiting a collective phenomenon of active rotors, our study paves the way to understanding chiral active matter.

cond-mat.soft↗

arXiv2021

Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narration

Given a piece of speech and its transcript text, text-based speech editing aims to generate speech that can be seamlessly inserted into the given speech by editing the transcript. Existing methods adopt a two-stage approach: synthesize the input text using a generic text-to-speech (TTS) engine and then transform the voice to the desired voice using voice conversion (VC). A major problem of this framework is that VC is a challenging problem which usually needs a moderate amount of parallel training data to work satisfactorily. In this paper, we propose a one-stage context-aware framework to generate natural and coherent target speech without any training data of the target speaker. In particular, we manage to perform accurate zero-shot duration prediction for the inserted text. The predicted duration is used to regulate both text embedding and speech embedding. Then, based on the aligned cross-modality input, we directly generate the mel-spectrogram of the edited speech with a transformer-based decoder. Subjective listening tests show that despite the lack of training data for the speaker, our method has achieved satisfactory results. It outperforms a recent zero-shot TTS engine by a large margin.

cs.SD↗

arXiv2021

VisDrone-CC2020: The Vision Meets Drone Crowd Counting Challenge Results

Crowd counting on the drone platform is an interesting topic in computer vision, which brings new challenges such as small object inference, background clutter and wide viewpoint. However, there are few algorithms focusing on crowd counting on the drone-captured data due to the lack of comprehensive datasets. To this end, we collect a large-scale dataset and organize the Vision Meets Drone Crowd Counting Challenge (VisDrone-CC2020) in conjunction with the 16th European Conference on Computer Vision (ECCV 2020) to promote the developments in the related fields. The collected dataset is formed by $3,360$ images, including $2,460$ images for training, and $900$ images for testing. Specifically, we manually annotate persons with points in each video frame. There are $14$ algorithms from $15$ institutes submitted to the VisDrone-CC2020 Challenge. We provide a detailed analysis of the evaluation results and conclude the challenge. More information can be found at the website: \url{http://www.aiskyeye.com/}.

cs.CV↗

arXiv2021

Identity Test of Single NV$^-$ Centers in Diamond at Hz-Precision Level

Atomic-like defects in solids are not considered to be identical owing to the imperfections of host lattice. Here, we found that even under ambient conditions, negatively charged nitrogen-vacancy (NV$^-$) centers in diamond could still manifest identical at Hz-precision level, corresponding to a 10$^{-7}$-level relative precision, while the lattice strain can destroy the identity by tens of Hz. All parameters involved in the NV$^-$-$^{14}$N Hamiltonian are determined by formulating six nuclear frequencies at 10-mHz-level precision and measuring them at Hz-level precision. The most precisely measured parameter, the $^{14}$N quadrupole coupling $P$, is given by -4945754.9(8) Hz, whose precision is improved by nearly four orders of magnitude compared with previous measurements. We offer an approach for performing precision measurements in solids and deepening our understandings of NV centers as well as other solid-state defects. Besides, these high-precision results imply a potential application of a robust and integrated atomic-like clock based on ensemble NV centers.

quant-ph↗