arXiv subjects

Ziyang Zhang

Publications and source records attributed to Ziyang Zhang.

106 records · Page 6Linked to original sources

arXiv2022

Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis

In this paper, we present our solutions for the Multimodal Sentiment Analysis Challenge (MuSe) 2022, which includes MuSe-Humor, MuSe-Reaction and MuSe-Stress Sub-challenges. The MuSe 2022 focuses on humor detection, emotional reactions and multimodal emotional stress utilizing different modalities and data sets. In our work, different kinds of multimodal features are extracted, including acoustic, visual, text and biological features. These features are fused by TEMMA and GRU with self-attention mechanism frameworks. In this paper, 1) several new audio features, facial expression features and paragraph-level text embeddings are extracted for accuracy improvement. 2) we substantially improve the accuracy and reliability of multimodal sentiment prediction by mining and blending the multimodal features. 3) effective data augmentation strategies are applied in model training to alleviate the problem of sample imbalance and prevent the model from learning biased subject characters. For the MuSe-Humor sub-challenge, our model obtains the AUC score of 0.8932. For the MuSe-Reaction sub-challenge, the Pearson's Correlations Coefficient of our approach on the test set is 0.3879, which outperforms all other participants. For the MuSe-Stress sub-challenge, our approach outperforms the baseline in both arousal and valence on the test dataset, reaching a final combined result of 0.5151.

cs.CV↗

arXiv2022

Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs

Model hubs with many pre-trained models (PTMs) have become a cornerstone of deep learning. Although built at a high cost, they remain \emph{under-exploited} -- practitioners usually pick one PTM from the provided model hub by popularity and then fine-tune the PTM to solve the target task. This naïve but common practice poses two obstacles to full exploitation of pre-trained model hubs: first, the PTM selection by popularity has no optimality guarantee, and second, only one PTM is used while the remaining PTMs are ignored. An alternative might be to consider all possible combinations of PTMs and extensively fine-tune each combination, but this would not only be prohibitive computationally but may also lead to statistical over-fitting. In this paper, we propose a new paradigm for exploiting model hubs that is intermediate between these extremes. The paradigm is characterized by two aspects: (1) We use an evidence maximization procedure to estimate the maximum value of label evidence given features extracted by pre-trained models. This procedure can rank all the PTMs in a model hub for various types of PTMs and tasks \emph{before fine-tuning}. (2) The best ranked PTM can either be fine-tuned and deployed if we have no preference for the model's architecture or the target PTM can be tuned by the top $K$ ranked PTMs via a Bayesian procedure that we propose. This procedure, which we refer to as \emph{B-Tuning}, not only improves upon specialized methods designed for tuning homogeneous PTMs, but also applies to the challenging problem of tuning heterogeneous PTMs where it yields a new level of benchmark performance.

cs.LG↗

arXiv2022

Incorporating Voice Instructions in Model-Based Reinforcement Learning for Self-Driving Cars

This paper presents a novel approach that supports natural language voice instructions to guide deep reinforcement learning (DRL) algorithms when training self-driving cars. DRL methods are popular approaches for autonomous vehicle (AV) agents. However, most existing methods are sample- and time-inefficient and lack a natural communication channel with the human expert. In this paper, how new human drivers learn from human coaches motivates us to study new ways of human-in-the-loop learning and a more natural and approachable training interface for the agents. We propose incorporating natural language voice instructions (NLI) in model-based deep reinforcement learning to train self-driving cars. We evaluate the proposed method together with a few state-of-the-art DRL methods in the CARLA simulator. The results show that NLI can help ease the training process and significantly boost the agents' learning speed.

cs.HC↗

arXiv2022

Fiber spectrum analyzer based on planar waveguide array aligned to a camera without lens

We propose and experimentally demonstrate a fiber spectrum analyzer based on a planar waveguide chip butt-coupled with an input fiber and aligned to a standard camera without any free-space optical elements. The chip consists of a single-mode waveguide to connect with the fiber, a beam broadening area, and a waveguide array in which the lengths of the waveguides are designed for both wavelength separation and beam focusing. The facet of the chip is diced open so that the outputs of the array form a near-field emitter. The far field are calculated by the Rayleigh-Sommerfeld diffraction integral. We show that the chip can provide a focal depth on the millimeter scale, allowing relaxed alignment to the camera without any fine-positioning stage. Two devices with 120 and 220 waveguides are fabricated on the polymer waveguide platform. The measured spectral width are 0.63 nm and 0.42 nm, respectively. This simple and practical approach may lead to the development of a spectrum analyzer for fiber that is easily mountable to any commercial camera, thereby avoiding the complication for customized detectors as well as electronic circuits afterwards.

physics.optics↗

arXiv2022

A unified theory of information transfer and causal relation

Information transfer between coupled stochastic dynamics, measured by transfer entropy and information flow, is suggested as a physical process underlying the causal relation of systems. While information transfer analysis has booming applications in both science and engineering fields, critical mysteries about its foundations remain unsolved. Fundamental yet difficult questions concern how information transfer and causal relation originate, what they depend on, how they differ from each other, and if they are created by a unified and general quantity. These questions essentially determine the validity of causal relation measurement via information transfer. Here we pursue to lay a complete theoretical basis of information transfer and causal relation. Beyond the well-known relations between these concepts that conditionally hold, we demonstrate that information transfer and causal relation universally originate from specific information synergy and redundancy phenomena characterized by high-order mutual information. More importantly, our theory analytically explains the mechanisms for information transfer and causal relation to originate, vanish, and differ from each other. Moreover, our theory naturally defines the effect sizes of information transfer and causal relation based on high-dimensional coupling events. These results may provide a unified view of information, synergy, and causal relation to bridge Pearl's causal inference theory in computer science and information transfer analysis in physics.

cond-mat.stat-mech↗

arXiv2022

Brain-inspired Multilayer Perceptron with Spiking Neurons

Recently, Multilayer Perceptron (MLP) becomes the hotspot in the field of computer vision tasks. Without inductive bias, MLPs perform well on feature extraction and achieve amazing results. However, due to the simplicity of their structures, the performance highly depends on the local features communication machenism. To further improve the performance of MLP, we introduce information communication mechanisms from brain-inspired neural networks. Spiking Neural Network (SNN) is the most famous brain-inspired neural network, and achieve great success on dealing with sparse data. Leaky Integrate and Fire (LIF) neurons in SNNs are used to communicate between different time steps. In this paper, we incorporate the machanism of LIF neurons into the MLP models, to achieve better accuracy without extra FLOPs. We propose a full-precision LIF operation to communicate between patches, including horizontal LIF and vertical LIF in different directions. We also propose to use group LIF to extract better local features. With LIF modules, our SNN-MLP model achieves 81.9%, 83.3% and 83.5% top-1 accuracy on ImageNet dataset with only 4.4G, 8.5G and 15.2G FLOPs, respectively, which are state-of-the-art results as far as we know.

cs.CV↗

arXiv2022

TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation

Recording fast motion in a high FPS (frame-per-second) requires expensive high-speed cameras. As an alternative, interpolating low-FPS videos from commodity cameras has attracted significant attention. If only low-FPS videos are available, motion assumptions (linear or quadratic) are necessary to infer intermediate frames, which fail to model complex motions. Event camera, a new camera with pixels producing events of brightness change at the temporal resolution of $μs$ $(10^{-6}$ second $)$, is a game-changing device to enable video interpolation at the presence of arbitrarily complex motion. Since event camera is a novel sensor, its potential has not been fulfilled due to the lack of processing algorithms. The pioneering work Time Lens introduced event cameras to video interpolation by designing optical devices to collect a large amount of paired training data of high-speed frames and events, which is too costly to scale. To fully unlock the potential of event cameras, this paper proposes a novel TimeReplayer algorithm to interpolate videos captured by commodity cameras with events. It is trained in an unsupervised cycle-consistent style, canceling the necessity of high-speed training data and bringing the additional ability of video extrapolation. Its state-of-the-art results and demo videos in supplementary reveal the promising future of event-based vision.

cs.CV↗

arXiv2022

MDNet: Learning Monaural Speech Enhancement from Deep Prior Gradient

While traditional statistical signal processing model-based methods can derive the optimal estimators relying on specific statistical assumptions, current learning-based methods further promote the performance upper bound via deep neural networks but at the expense of high encapsulation and lack adequate interpretability. Standing upon the intersection between traditional model-based methods and learning-based methods, we propose a model-driven approach based on the maximum a posteriori (MAP) framework, termed as MDNet, for single-channel speech enhancement. Specifically, the original problem is formulated into the joint posterior estimation w.r.t. speech and noise components. Different from the manual assumption toward the prior terms, we propose to model the prior distribution via networks and thus can learn from training data. The framework takes the unfolding structure and in each step, the target parameters can be progressively estimated through explicit gradient descent operations. Besides, another network serves as the fusion module to further refine the previous speech estimation. The experiments are conducted on the WSJ0-SI84 and Interspeech2020 DNS-Challenge datasets, and quantitative results show that the proposed approach outshines previous state-of-the-art baselines.

cs.SD↗

arXiv2022

From Big to Small: Adaptive Learning to Partial-Set Domains

Domain adaptation targets at knowledge acquisition and dissemination from a labeled source domain to an unlabeled target domain under distribution shift. Still, the common requirement of identical class space shared across domains hinders applications of domain adaptation to partial-set domains. Recent advances show that deep pre-trained models of large scale endow rich knowledge to tackle diverse downstream tasks of small scale. Thus, there is a strong incentive to adapt models from large-scale domains to small-scale domains. This paper introduces Partial Domain Adaptation (PDA), a learning paradigm that relaxes the identical class space assumption to that the source class space subsumes the target class space. First, we present a theoretical analysis of partial domain adaptation, which uncovers the importance of estimating the transferable probability of each class and each instance across domains. Then, we propose Selective Adversarial Network (SAN and SAN++) with a bi-level selection strategy and an adversarial adaptation mechanism. The bi-level selection strategy up-weighs each class and each instance simultaneously for source supervised training, target self-training, and source-target adversarial adaptation through the transferable probability estimated alternately by the model. Experiments on standard partial-set datasets and more challenging tasks with superclasses show that SAN++ outperforms several domain adaptation methods.

cs.LG↗

arXiv2021

Fourier-domain transfer entropy spectrum

We propose the Fourier-domain transfer entropy spectrum, a novel generalization of transfer entropy, as a model-free metric of causality. For arbitrary systems, this approach systematically quantifies the causality among their different system components rather than merely analyze systems as entireties. The generated spectrum offers a rich-information representation of time-varying latent causal relations, efficiently dealing with non-stationary processes and high-dimensional conditions. We demonstrate its validity in the aspects of parameter dependence, statistic significance test, and sensibility. An open-source multi-platform implementation of this metric is developed and computationally applied on neuroscience data sets and diffusively coupled logistic oscillators.

physics.data-an↗

arXiv2020

Deep Spatial Learning with Molecular Vibration

Machine learning over-fitting caused by data scarcity greatly limits the application of machine learning for molecules. Due to manufacturing processes difference, big data is not always rendered available through computational chemistry methods for some tasks, causing data scarcity problem for machine learning algorithms. Here we propose to extract the natural features of molecular structures and rationally distort them to augment the data availability. This method allows a machine learning project to leverage the powerful fit of physics-informed augmentation for providing significant boost to predictive accuracy. Successfully verified by the prediction of rejection rate and flux of thin film polyamide nanofiltration membranes, with the relative error dropping from 16.34% to 6.71% and the coefficient of determination rising from 0.16 to 0.75, the proposed deep spatial learning with molecular vibration is widely instructive for molecular science. Experimental comparison unequivocally demonstrates its superiority over common learning algorithms.

cs.LG↗

arXiv2019

Stochastic Sparse Learning with Momentum Adaptation for Imprecise Memristor Networks

Memristor based neural networks have great potentials in on-chip neuromorphic computing systems due to the fast computation and low-energy consumption. However, the imprecise properties of existing memristor devices generally result in catastrophic failures for the network in-situ training, which significantly impedes their engineering applications. In this work, we design a novel learning scheme that integrates stochastic sparse updating with momentum adaption (SSM) to efficiently train the imprecise memristor networks with high classification accuracy. The SSM scheme consists of: (1) a stochastic and discrete learning method to make weight updates sparse; (2) a momentum based gradient algorithm to eliminate training noises and distill robust updates; (3) a network re-initialization method to mitigate the device-to-device variation; (4) an update compensation strategy to further stabilize the weight programming process. With the SSM scheme, experiments show that the classification accuracy on multilayer perceptron (MLP) and convolutional neural network (CNN) improves from 26.12% to 90.07% and from 65.98% to 92.38%, respectively. Meanwhile, the total numbers of weight updating pulses decrease 90% and 40% in MLP and CNN, respectively, and the convergence rates are both 3x faster. The SSM scheme provides a high-accuracy, low-power, and fast-convergence solution for the in-situ training of imprecise memristor networks, which is crucial to future neuromorphic intelligence systems.

cs.ET↗

arXiv2018

Aperiodic phase masks for inscribing complex multi-notch OH-emission filters for astronomy

We demonstrate for the first time, a new type of aperiodic phase mask (APM) for fabricating multi-channel aperiodic fiber Bragg gratings. The mask is made of individual diffraction phase gratings with discrete unequal phase-steps incorporated at periodic locations. The diffraction at the discrete phase-steps in the phase mask produces corresponding half phase-steps at periodic locations along the fiber. The accumulated phase, along with index modulation, generates the desired multinotch reflection spectrum. Complex fiber Bragg grating filters fabricated using APM, in a standard phase mask based fabrication setup, can be used to simultaneously suppress multiple aperiodic OH emission wavelengths in near infrared (NIR) existing in upper atmosphere, and increase the sensitivity of ground based telescopes.

astro-ph.IM↗

arXiv2018

Truly concomitant and independently expressed short- and long-term plasticity in Bi2O2Se-based three-terminal memristor

Orchestration of diverse synaptic plasticity mechanisms across different timescales produces complex cognitive processes. To achieve comparable cognitive complexity in memristive neuromorphic systems, devices that are capable to emulate short- and long-term plasticity (STP and LTP, respectively) concomitantly are essential. However, this fundamental bionic trait has not been reported in any existing memristors where STP and LTP can only be induced selectively because of the inability to be decoupled using different loci and mechanisms. In this work, we report the first demonstration of truly concomitant STP and LTP in a three-terminal memristor that uses independent physical phenomena to represent each form of plasticity. The emerging layered material Bi2O2Se is used in memristor for the first time, opening up the prospects for ultra-thin, high-speed and low-power neuromorphic devices. The concerted action of STP and LTP in our memristor allows full-range modulation of the transient synaptic efficacy, from depression to facilitation, by stimulus frequency or intensity, providing a versatile device platform for neuromorphic function implementation. A recurrent neural circuitry model is developed to simulate the intricate "sleep-wake cycle autoregulation" process, in which the concomitance of STP and LTP is posited as a key factor in enabling this neural homeostasis. This work sheds new light on the highly sophisticated computational capabilities of memristors and their prospects for realization of advanced neuromorphic functions.

physics.app-ph↗

arXiv2015

The electronic structures of TiO2/Ti4O7, Ta2O5/TaO2 interfaces and the interfacial effects of dopants

We study the electronic structures of TiO2/Ti4O7 and Ta2O5/TaO2 interfaces using the screened exchange (sX-LDA) functional. Both of these bilayer structures are useful infrastructures for high performance RRAMs. We find that the system Fermi energies of both interfaces are just above the conduction band edge of the corresponding stoichiometric oxides. According to the charge transition levels of the oxygen vacancies in Ta2O5 and TiO2, the oxygen vacancies are stabilized at the -2 charged state. We propose to introduce interfacial dopants to shift the system Fermi energies downward so that the +2 charged oxygen vacancy can be stable, which is important for the controlled switching under the electrical field. Several dipole models are presented to account for the ability of Fermi level shift due to the interfacial dopants.

cond-mat.mtrl-sci↗

arXiv2015

The role of single oxygen or metal induced defect and correlated multiple defects in the formation of conducting filaments

We study the dependence of the formation energies of oxygen and metal induced defects in Ta2O5, TaO2, TaO, TiO2 and Ti4O7 on the chemical potential of electron and atomic constitutes. In the study of single defect, metal induced defects are found to be preferable to oxygen induced defects. This is against the experimental fact of the dominant role of oxygen induced defects in the RS process. A simple multiple defects picture without correlated atomic rearrangement does not cure this problem. The problem is resolved under the correlated multiple defect picture where the multiple defects result in correlated atomic rearrangement and the final products show certain atomic ordering.

cond-mat.mtrl-sci↗