Searcharxiv⌕ Search

arXiv subjects

Zhao Chen

Publications and source records attributed to Zhao Chen.

At least 55 records · Page 3Linked to original sources
arXiv2022

HyperPrompt: Prompt-based Task-Conditioning of Transformers

Prompt-Tuning is a new paradigm for finetuning pre-trained language models in a parameter-efficient way. Here, we explore the use of HyperNetworks to generate hyper-prompts: we propose HyperPrompt, a novel architecture for prompt-based task-conditioning of self-attention in Transformers. The hyper-prompts are end-to-end learnable via generation by a HyperNetwork. HyperPrompt allows the network to learn task-specific feature maps where the hyper-prompts serve as task global memories for the queries to attend to, at the same time enabling flexible information sharing among tasks. We show that HyperPrompt is competitive against strong multi-task learning baselines with as few as $0.14\%$ of additional task-conditioning parameters, achieving great parameter and computational efficiency. Through extensive empirical experiments, we demonstrate that HyperPrompt can achieve superior performances over strong T5 multi-task learning baselines and parameter-efficient adapter variants including Prompt-Tuning and HyperFormer++ on Natural Language Understanding benchmarks of GLUE and SuperGLUE across many model sizes.

cs.CL↗
arXiv2022

GradTail: Learning Long-Tailed Data Using Gradient-based Sample Weighting

We propose GradTail, an algorithm that uses gradients to improve model performance on the fly in the face of long-tailed training data distributions. Unlike conventional long-tail classifiers which operate on converged - and possibly overfit - models, we demonstrate that an approach based on gradient dot product agreement can isolate long-tailed data early on during model training and improve performance by dynamically picking higher sample weights for that data. We show that such upweighting leads to model improvements for both classification and regression models, the latter of which are relatively unexplored in the long-tail literature, and that the long-tail examples found by gradient alignment are consistent with our semantic expectations.

cs.LG↗
arXiv2021

State-resolved ultrafast charge and spin dynamics in [Co/Pd] multilayers

We use transient absorption spectroscopy with circularly polarized x-rays to detect laser-excited hole states below the Fermi level and compare their dynamics with that of unoccupied states above the Fermi level in ferromagnetic [Co/Pd] multilayers. While below the Fermi level an instantaneous and significantly stronger demagnetization is observed, above the Fermi level the demagnetization is delayed by 35+/-10 fs. This provides a direct visualization of how ultrafast demagnetization proceeds via initial spin-flip scattering of laser-excited holes to the subsequent formation of spin waves.

cond-mat.mtrl-sci↗
arXiv2021

Taskology: Utilizing Task Relations at Scale

Many computer vision tasks address the problem of scene understanding and are naturally interrelated e.g. object classification, detection, scene segmentation, depth estimation, etc. We show that we can leverage the inherent relationships among collections of tasks, as they are trained jointly, supervising each other through their known relationships via consistency losses. Furthermore, explicitly utilizing the relationships between tasks allows improving their performance while dramatically reducing the need for labeled data, and allows training with additional unsupervised or simulated data. We demonstrate a distributed joint training algorithm with task-level parallelism, which affords a high degree of asynchronicity and robustness. This allows learning across multiple tasks, or with large amounts of input data, at scale. We demonstrate our framework on subsets of the following collection of tasks: depth and normal prediction, semantic segmentation, 3D motion and ego-motion estimation, and object tracking and 3D detection in point clouds. We observe improved performance across these tasks, especially in the low-label regime.

cs.CV↗
arXiv2021

Physics-informed learning of governing equations from scarce data

Harnessing data to discover the underlying governing laws or equations that describe the behavior of complex physical systems can significantly advance our modeling, simulation and understanding of such systems in various science and engineering disciplines. This work introduces a novel physics-informed deep learning framework to discover governing partial differential equations (PDEs) from scarce and noisy data for nonlinear spatiotemporal systems. In particular, this approach seamlessly integrates the strengths of deep neural networks for rich representation learning, physics embedding, automatic differentiation and sparse regression to (1) approximate the solution of system variables, (2) compute essential derivatives, as well as (3) identify the key derivative terms and parameters that form the structure and explicit expression of the PDEs. The efficacy and robustness of this method are demonstrated, both numerically and experimentally, on discovering a variety of PDE systems with different levels of data scarcity and noise accounting for different initial/boundary conditions. The resulting computational framework shows the potential for closed-form model discovery in practical applications where large and accurate datasets are intractable to capture.

cs.LG↗
arXiv2020

Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout

The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights. However, these multiple updates can impede optimal training by pulling the model in conflicting directions. We present Gradient Sign Dropout (GradDrop), a probabilistic masking procedure which samples gradients at an activation layer based on their level of consistency. GradDrop is implemented as a simple deep layer that can be used in any deep net and synergizes with other gradient balancing approaches. We show that GradDrop outperforms the state-of-the-art multiloss methods within traditional multitask and transfer learning settings, and we discuss how GradDrop reveals links between optimal multiloss training and gradient stochasticity.

cs.LG↗
arXiv2020

Sparse representation for damage identification of structural systems

Identifying damage of structural systems is typically characterized as an inverse problem which might be ill-conditioned due to aleatory and epistemic uncertainties induced by measurement noise and modeling error. Sparse representation can be used to perform inverse analysis for the case of sparse damage. In this paper, we propose a novel two-stage sensitivity analysis-based framework for both model updating and sparse damage identification. Specifically, an $\ell_2$ Bayesian learning method is firstly developed for updating the intact model and uncertainty quantification so as to set forward a baseline for damage detection. A sparse representation pipeline built on a quasi-$\ell_0$ method, e.g., Sequential Threshold Least Squares (STLS) regression, is then presented for damage localization and quantification. Additionally, Bayesian optimization together with cross validation is developed to heuristically learn hyperparameters from data, which saves the computational cost of hyperparameter tuning and produces more reliable identification result. The proposed framework is verified by three examples, including a 10-story shear-type building, a complex truss structure, and a shake table test of an eight-story steel frame. Results show that the proposed approach is capable of both localizing and quantifying structural damage with high accuracy.

stat.AP↗
arXiv2020

The Weak Lensing Peak Statistics in the Mocks by the inverse-Gaussianization Method

We apply the inverse-Gaussianization method proposed in \citealt{arXiv:1607.05007} to fast produce weak lensing convergence maps and investigate the peak statistics, including the peak height counts and peak steepness counts, in these mocks. We find that the distribution of peak height and steepness is in good agreement with the simulation. The difference is $\lesssim 20\%$ for these peak statistics in the maps at source redshift $z_s=1$. Besides, the loss of off-diagonal elements in peak covariance motivates us to consider the super sample variance in weak lensing peak statistics. We propose correction methods to effectively recover the (anti-)correlation among different bins by adding scatters in the mean value of these mocks. Finally, as an example of the application, we adopt the improved inverse-Gaussianization method with the above improvement to fast generate 40,000 mocks to calculate precision matrices between the power spectrum and peak statistics.

astro-ph.CO↗
arXiv2019

Ranking Users in Social Networks with Motif-based PageRank

PageRank has been widely used to measure the authority or the influence of a user in social networks. However, conventional PageRank only makes use of edge-based relations, which represent first-order relations between two connected nodes. It ignores higher-order relations that may exist between nodes. In this paper, we propose a novel framework, motif-based PageRank (MPR), to incorporate higher-order relations into the conventional PageRank computation. Motifs are subgraphs consisting of a small number of nodes. We use motifs to capture higher-order relations between nodes in a network and introduce two methods, one linear and one non-linear, to combine PageRank with higher-order relations. We conduct extensive experiments on three real-world networks, namely, DBLP, Epinions, and Ciao. We study different types of motifs, including 3-node simple and anchor motifs, 4-node and 5-node motifs. Besides using single motif, we also run MPR with ensemble of multiple motifs. We also design a learning task to evaluate the abilities of authority prediction with motif-based features. All experimental results demonstrate that MPR can significantly improve the performance of user ranking in social networks compared to the baseline methods.

cs.SI↗
arXiv2019

DeepPerimeter: Indoor Boundary Estimation from Posed Monocular Sequences

We present DeepPerimeter, a deep learning based pipeline for inferring a full indoor perimeter (i.e. exterior boundary map) from a sequence of posed RGB images. Our method relies on robust deep methods for depth estimation and wall segmentation to generate an exterior boundary point cloud, and then uses deep unsupervised clustering to fit wall planes to obtain a final boundary map of the room. We demonstrate that DeepPerimeter results in excellent visual and quantitative performance on the popular ScanNet and FloorNet datasets and works for room shapes of various complexities as well as in multiroom scenarios. We also establish important baselines for future work on indoor perimeter estimation, topics which will become increasingly prevalent as application areas like augmented reality and robotics become more significant.

cs.CV↗
arXiv2019

Distinguishing Local and non-Local Demagnetization in Ferromagnetic FePt Nanoparticles

Time-resolved coherent X-ray diffraction is used to measure the spatially resolved magnetization structure within FePt nanoparticles during laser-induced ultrafast demagnetization. The momentum-dependent X-ray magnetic diffraction shows that demagnetization proceeds at different rates at different X-ray momentum transfer. We show that the observed momentum-dependent scattering has the signature of inhomogeneous demagnetization within the nanoparticles, with the demagnetization proceeding more rapidly at the boundary of the nanoparticle. A shell region of reduced magnetization forms and moves inwards at a supermagnonic velocity. Spin-transport calculations show that the shell formation is driven by superdiffusive spin flux mainly leaving the nanoparticle into the surrounding carbon. Quantifying this non-local contribution to the demagnetization allows us to separate it from the local demagnetization.

cond-mat.mtrl-sci↗
arXiv2019

Ultrafast X-Ray Induced Changes of the Electronic and Magnetic Response of Solids Due to Valence Electron Redistribution

We report a novel mechanism, consisting of redistribution of valence electrons near the Fermi level, during interactions of intense femtosecond X-ray pulses with a Co/Pd multilayer. The changes in Co 3d valence shell occupation were directly revealed by fluence-dependent changes of the Co L$_3$ X-ray absorption and magnetic circular dichroism spectra near the excitation threshold. The valence shell redistribution arises from inelastic scattering of high energy Auger electrons and photoelectrons that lead to transient holes below and electrons above the Fermi level on the femtosecond time scale. The valence electron reshuffling effect scales with the energy deposited by X-rays and within 17 fs extends to valence states within 2 eV of the Fermi level. As a consequence the sample demagnetizes by more than twenty percent due to magnon generation.

cond-mat.mtrl-sci↗
arXiv2018

Decentralized Computation Offloading for Multi-User Mobile Edge Computing: A Deep Reinforcement Learning Approach

Mobile edge computing (MEC) emerges recently as a promising solution to relieve resource-limited mobile devices from computation-intensive tasks, which enables devices to offload workloads to nearby MEC servers and improve the quality of computation experience. Nevertheless, by considering a MEC system consisting of multiple mobile users with stochastic task arrivals and wireless channels in this paper, the design of computation offloading policies is challenging to minimize the long-term average computation cost in terms of power consumption and buffering delay. A deep reinforcement learning (DRL) based decentralized dynamic computation offloading strategy is investigated to build a scalable MEC system with limited feedback. Specifically, a continuous action space-based DRL approach named deep deterministic policy gradient (DDPG) is adopted to learn efficient computation offloading policies independently at each mobile user. Thus, powers of both local execution and task offloading can be adaptively allocated by the learned policies from each user's local observation of the MEC system. Numerical results are illustrated to demonstrate that efficient policies can be learned at each user, and performance of the proposed DDPG based decentralized strategy outperforms the conventional deep Q-network (DQN) based discrete power control strategy and some other greedy strategies with reduced computation cost. Besides, the power-delay tradeoff is also analyzed for both the DDPG based and DQN based strategies.

cs.LG↗
arXiv2018

Estimating Depth from RGB and Sparse Sensing

We present a deep model that can accurately produce dense depth maps given an RGB image with known depth at a very sparse set of pixels. The model works simultaneously for both indoor/outdoor scenes and produces state-of-the-art dense depth maps at nearly real-time speeds on both the NYUv2 and KITTI datasets. We surpass the state-of-the-art for monocular depth estimation even with depth values for only 1 out of every ~10000 image pixels, and we outperform other sparse-to-dense depth methods at all sparsity levels. With depth values for 1/256 of the image pixels, we achieve a mean absolute error of less than 1% of actual depth on indoor scenes, comparable to the performance of consumer-grade depth sensor hardware. Our experiments demonstrate that it would indeed be possible to efficiently transform sparse depth measurements obtained using e.g. lower-power depth sensors or SLAM systems into high-quality dense depth maps.

cs.CV↗
arXiv2018

Gradient Adversarial Training of Neural Networks

We propose gradient adversarial training, an auxiliary deep learning framework applicable to different machine learning problems. In gradient adversarial training, we leverage a prior belief that in many contexts, simultaneous gradient updates should be statistically indistinguishable from each other. We enforce this consistency using an auxiliary network that classifies the origin of the gradient tensor, and the main network serves as an adversary to the auxiliary network in addition to performing standard task-based training. We demonstrate gradient adversarial training for three different scenarios: (1) as a defense to adversarial examples we classify gradient tensors and tune them to be agnostic to the class of their corresponding example, (2) for knowledge distillation, we do binary classification of gradient tensors derived from the student or teacher network and tune the student gradient tensor to mimic the teacher's gradient tensor; and (3) for multi-task learning we classify the gradient tensors derived from different task loss functions and tune them to be statistically indistinguishable. For each of the three scenarios we show the potential of gradient adversarial training procedure. Specifically, gradient adversarial training increases the robustness of a network to adversarial attacks, is able to better distill the knowledge from a teacher network to a student network compared to soft targets, and boosts multi-task learning by aligning the gradient tensors derived from the task specific loss functions. Overall, our experiments demonstrate that gradient tensors contain latent information about whatever tasks are being trained, and can support diverse machine learning problems when intelligently guided through adversarialization using a auxiliary network.

cs.LG↗
arXiv2018

GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Deep multitask networks, in which one neural network produces multiple predictive outputs, can offer better speed and performance than their single-task counterparts but are challenging to train properly. We present a gradient normalization (GradNorm) algorithm that automatically balances training in deep multitask models by dynamically tuning gradient magnitudes. We show that for various network architectures, for both regression and classification tasks, and on both synthetic and real datasets, GradNorm improves accuracy and reduces overfitting across multiple tasks when compared to single-task networks, static baselines, and other adaptive multitask loss balancing techniques. GradNorm also matches or surpasses the performance of exhaustive grid search methods, despite only involving a single asymmetry hyperparameter $α$. Thus, what was once a tedious search process that incurred exponentially more compute for each task added can now be accomplished within a few training runs, irrespective of the number of tasks. Ultimately, we will demonstrate that gradient manipulation affords us great control over the training dynamics of multitask networks and may be one of the keys to unlocking the potential of multitask learning.

cs.CV↗
arXiv2018

A Stochastic Geometry Analysis of Energy Harvesting in Large Scale Wireless Networks

In this paper, the theoretical sustainable capacity of wireless networks with radio frequency (RF) energy harvesting is analytically studied. Specifically, we consider a large scale wireless network where base stations (BSs) and low power wireless devices are deployed by homogeneous Poisson point process (PPP) with different spatial densities. Wireless devices exploit the downlink transmissions from the BSs for either information delivery or energy harvesting. Generally, a BS schedules downlink transmission to wireless devices. The scheduled device receives the data information while other devices harvest energy from the downlink signals. The data information can be successfully received by the scheduled device only if the device has sufficient energy for data processing, i.e., the harvested energy is larger than a threshold. Given the densities of BSs and users, we apply stochastic geometry to analyze the expected number of users per cell and the successful information delivery probability of a wireless device, based on which the total network throughput can be derived. It is shown that the maximum network throughput per cell can be achieved under the optimal density of BSs. Extensive simulations validate the analysis.

cs.NI↗