arXiv subjects

Zhao Zhang

Publications and source records attributed to Zhao Zhang.

At least 235 records · Page 13Linked to original sources

arXiv2021

KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks

Kronecker-factored Approximate Curvature (K-FAC) has recently been shown to converge faster in deep neural network (DNN) training than stochastic gradient descent (SGD); however, K-FAC's larger memory footprint hinders its applicability to large models. We present KAISA, a K-FAC-enabled, Adaptable, Improved, and ScAlable second-order optimizer framework that adapts the memory footprint, communication, and computation given specific models and hardware to improve performance and increase scalability. We quantify the tradeoffs between memory and communication cost and evaluate KAISA on large models, including ResNet-50, Mask R-CNN, U-Net, and BERT, on up to 128 NVIDIA A100 GPUs. Compared to the original optimizers, KAISA converges 18.1-36.3% faster across applications with the same global batch size. Under a fixed memory budget, KAISA converges 32.5% and 41.6% faster in ResNet-50 and BERT-Large, respectively. KAISA can balance memory and communication to achieve scaling efficiency equal to or better than the baseline optimizers. KAISA is open source and available at https://github.com/gpauloski/kfac_pytorch.

cs.LG↗

arXiv2021

Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior

Nonnegative matrix factorization is usually powerful for learning the "shallow" parts-based representation, but it clearly fails to discover deep hierarchical information within both the basis and representation spaces. In this paper, we technically propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net, for learning the hierarchical coupled representations. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Specifically, DS2CF-Net designs a deep coupled factorization architecture using multi-layers of linear transformations, which coupled updates the bases and new representations in each layer. To improve the discriminating ability of learned deep representations and deep coefficients, our network clearly considers enriching the supervised prior by the joint deep coefficients-regularized label prediction, and incorporates enriched prior information as additional label and structure constraints. The label constraint can enable the samples of the same label to have the same coordinate in the new feature space, while the structure constraint forces the coefficient matrices in each layer to be block-diagonal so that the enhanced prior using the self-expressive label propagation are more accurate. Our network also integrates the adaptive dual-graph learning to retain the local manifold structures of both the data manifold and feature manifold by minimizing the reconstruction errors in each layer. Extensive experiments on several real databases demonstrate that our DS2CF-Net can obtain state-of-the-art performance for representation learning and clustering.

cs.LG↗

arXiv2021

Characterizing Impacts of Storage Faults on HPC Applications: A Methodology and Insights

In recent years, the increasing complexity in scientific simulations and emerging demands for training heavy artificial intelligence models require massive and fast data accesses, which urges high-performance computing (HPC) platforms to equip with more advanced storage infrastructures such as solid-state disks (SSDs). While SSDs offer high-performance I/O, the reliability challenges faced by the HPC applications under the SSD-related failures remains unclear, in particular for failures resulting in data corruptions. The goal of this paper is to understand the impact of SSD-related faults on the behaviors of complex HPC applications. To this end, we propose FFIS, a FUSE-based fault injection framework that systematically introduces storage faults into the application layer to model the errors originated from SSDs. FFIS is able to plant different I/O related faults into the data returned from underlying file systems, which enables the investigation on the error resilience characteristics of the scientific file format. We demonstrate the use of FFIS with three representative real HPC applications, showing how each application reacts to the data corruptions, and provide insights on the error resilience of the widely adopted HDF5 file format for the HPC applications.

cs.DC↗

arXiv2021

Many-to-English Machine Translation Tools, Data, and Pretrained Models

While there are more than 7000 languages in the world, most translation research efforts have targeted a few high-resource languages. Commercial translation systems support only one hundred languages or fewer, and do not make these models available for transfer to low resource languages. In this work, we present useful tools for machine translation research: MTData, NLCodec, and RTG. We demonstrate their usefulness by creating a multilingual neural machine translation model capable of translating from 500 source languages to English. We make this multilingual model readily downloadable and usable as a service, or as a parent model for transfer-learning to even lower-resource languages.

cs.CL↗

arXiv2021

Barocaloric thermal batteries

Nowadays the world is facing a prominent paradox regarding thermal energy. The production of heat accounts for more than 50% of global final energy consumption while the waste heat potential analysis reveals that 72% of the global primary energy consumption is lost after conversion mainly in the form of heat. Towards global decarbonization, it is of vital importance to establish a solution to thermal energy utilization under full control. Here, we propose and realize an unprecedented concept -- barocaloric thermal batteries based on the inverse colossal barocaloric effect of NH4SCN. Thermal charging is initialized upon pressurization through an order-to-disorder phase transition below 364 K and in turn the discharging of 43 J g-1, which are eleven times more than the input mechanical energy, occurs on demand at depressurization at lower temperatures. The discharging is also manifested as a directly measured temperature rise of 12 K. The thermodynamic equilibrium nature of the pressure-restrained heat-carrying phase guarantees stable storage and/or transport over a variety of temporal and/or spatial scales. The barocaloric thermal batteries reinforced by their solid microscopic mechanism are expected to significantly advance the ability to take advantage of waste heat.

cond-mat.mtrl-sci↗

arXiv2021

Phase transition gravitational waves from pseudo-Nambu-Goldstone dark matter and two Higgs doublets

We investigate the potential stochastic gravitational waves from first-order electroweak phase transitions in a model with pseudo-Nambu-Goldstone dark matter and two Higgs doublets. The dark matter candidate can naturally evade direct detection bounds, and can achieve the observed relic abundance via the thermal mechanism. Three scalar fields in the model obtain vacuum expectation values, related to phase transitions at the early Universe. We search for the parameter points that can cause first-order phase transitions, taking into account the existed experimental constraints. The resulting gravitational wave spectra are further evaluated. Some parameter points are found to induce strong gravitational wave signals, which have the opportunity to be detected in future space-based interferometer experiments LISA, Taiji, and TianQin.

hep-ph↗

arXiv2021

DerainCycleGAN: Rain Attentive CycleGAN for Single Image Deraining and Rainmaking

Single image deraining (SID) is an important and challenging topic in emerging vision applications, and most of emerged deraining methods are supervised relying on the ground truth (i.e., paired images) in recent years. However, in practice it is rather common to have no un-paired images in real deraining task, in such cases how to remove the rain streaks in an unsupervised way will be a very challenging task due to lack of constraints between images and hence suffering from low-quality recovery results. In this paper, we explore the unsupervised SID task using unpaired data and propose a novel net called Attention-guided Deraining by Constrained CycleGAN (or shortly, DerainCycleGAN), which can fully utilize the constrained transfer learning abilitiy and circulatory structure of CycleGAN. Specifically, we design an unsu-pervised attention guided rain streak extractor (U-ARSE) that utilizes a memory to extract the rain streak masks with two constrained cycle-consistency branches jointly by paying attention to both the rainy and rain-free image domains. As a by-product, we also contribute a new paired rain image dataset called Rain200A, which is constructed by our network automatically. Compared with existing synthesis datasets, the rainy streaks in Rain200A contains more obvious and diverse shapes and directions. As a result, existing supervised methods trained on Rain200A can perform much better for processing real rainy images. Extensive experiments on synthesis and real datasets show that our net is superior to existing unsupervised deraining networks, and is also very competitive to other related supervised networks.

cs.CV↗

arXiv2021

Semi-DerainGAN: A New Semi-supervised Single Image Deraining Network

Removing the rain streaks from single image is still a challenging task, since the shapes and directions of rain streaks in the synthetic datasets are very different from real images. Although supervised deep deraining networks have obtained impressive results on synthetic datasets, they still cannot obtain satisfactory results on real images due to weak generalization of rain removal capacity, i.e., the pre-trained models usually cannot handle new shapes and directions that may lead to over-derained/under-derained results. In this paper, we propose a new semi-supervised GAN-based deraining network termed Semi-DerainGAN, which can use both synthetic and real rainy images in a uniform network using two supervised and unsupervised processes. Specifically, a semi-supervised rain streak learner termed SSRML sharing the same parameters of both processes is derived, which makes the real images contribute more rain streak information. To deliver better deraining results, we design a paired discriminator for distinguishing the real pairs from fake pairs. Note that we also contribute a new real-world rainy image dataset Real200 to alleviate the difference between the synthetic and real image do-mains. Extensive results on public datasets show that our model can obtain competitive performance, especially on real images.

cs.CV↗

arXiv2021

Dense Residual Network: Enhancing Global Dense Feature Flow for Character Recognition

Deep Convolutional Neural Networks (CNNs), such as Dense Convolutional Networks (DenseNet), have achieved great success for image representation by discovering deep hierarchical information. However, most existing networks simply stacks the convolutional layers and hence failing to fully discover local and global feature information among layers. In this paper, we mainly explore how to enhance the local and global dense feature flow by exploiting hierarchical features fully from all the convolution layers. Technically, we propose an efficient and effective CNN framework, i.e., Fast Dense Residual Network (FDRN), for text recognition. To construct FDRN, we propose a new fast residual dense block (f-RDB) to retain the ability of local feature fusion and local residual learning of original RDB, which can reduce the computing efforts at the same time. After fully learning local residual dense features, we utilize the sum operation and several f-RDBs to define a new block termed global dense block (GDB) by imitating the construction of dense blocks to learn global dense residual features adaptively in a holistic way. Finally, we use two convolution layers to construct a down-sampling block to reduce the global feature size and extract deeper features. Extensive simulations show that FDRN obtains the enhanced recognition results, compared with other related models.

cs.CV↗

arXiv2021

A Survey on Concept Factorization: From Shallow to Deep Representation Learning

The quality of learned features by representation learning determines the performance of learning algorithms and the related application tasks (such as high-dimensional data clustering). As a relatively new paradigm for representation learning, Concept Factorization (CF) has attracted a great deal of interests in the areas of machine learning and data mining for over a decade. Lots of effective CF based methods have been proposed based on different perspectives and properties, but note that it still remains not easy to grasp the essential connections and figure out the underlying explanatory factors from exiting studies. In this paper, we therefore survey the recent advances on CF methodologies and the potential benchmarks by categorizing and summarizing the current methods. Specifically, we first re-view the root CF method, and then explore the advancement of CF-based representation learning ranging from shallow to deep/multilayer cases. We also introduce the potential application areas of CF-based methods. Finally, we point out some future directions for studying the CF-based representation learning. Overall, this survey provides an insightful overview of both theoretical basis and current developments in the field of CF, which can also help the interested researchers to understand the current trends of CF and find the most appropriate CF techniques to deal with particular applications.

cs.LG↗

arXiv2020

Prime Suspects in a Quantum Ladder

In this Letter we set up a suggestive number theory interpretation of a quantum ladder system made of N coupled chains of spin 1/2. Using the hard-core boson representation and a leg-Hamiltonian made of a magnetic field and a hopping term, we can associate to the spins $s_a$ the prime numbers $p_a$ so that the chains become quantum registers for square-free integers. The rung Hamiltonian involves permutation terms between next neighborhood chains and a coprime repulsive interaction. The system has various phases; in particular there is one whose ground state is a coherent superposition of the first N prime numbers. We also discuss the realization of such a model in terms of an open quantum system with a dissipative Lindblad dynamics.

cond-mat.stat-mech↗

arXiv2020

Gradient-Induced Co-Saliency Detection

Co-saliency detection (Co-SOD) aims to segment the common salient foreground in a group of relevant images. In this paper, inspired by human behavior, we propose a gradient-induced co-saliency detection (GICD) method. We first abstract a consensus representation for the grouped images in the embedding space; then, by comparing the single image with consensus representation, we utilize the feedback gradient information to induce more attention to the discriminative co-salient features. In addition, due to the lack of Co-SOD training data, we design a jigsaw training strategy, with which Co-SOD networks can be trained on general saliency datasets without extra pixel-level annotations. To evaluate the performance of Co-SOD methods on discovering the co-salient object among multiple foregrounds, we construct a challenging CoCA dataset, where each image contains at least one extraneous foreground along with the co-salient object. Experiments demonstrate that our GICD achieves state-of-the-art performance. Our codes and dataset are available at https://mmcheng.net/gicd/.

cs.CV↗

arXiv2020

Ultrafast ferromagnetic fluctuations preceding magnetoelastic first-order transitions

First-order magnetic transitions are of both fundamental and technological interest given that a number of emergent phases and functionalities are thereby created. Of particular interest are giant magnetocaloric effects, which are attributed to first-order magnetic transitions and have attracted broad attention for solid-state refrigeration applications. While the conventional wisdom is that atomic lattices play an important role in first-order magnetic transitions, a coherent microscopic description of the lattice and spin degrees of freedom is still lacking. Here, we study the magnetic phase transition dynamics on the intermetallic LaFe13-xSix, which is one of the most classical giant magnetocaloric systems, in both frequency and time domains utilizing neutron scattering and ultrafast X-ray diffraction. We have observed a strong magnetic diffuse scattering in the paramagnetic state preceding the first-order magnetic transition, corresponding to picosecond ferromagnetic fluctuations. Upon photon-excitation, the ferromagnetic state is completely suppressed in 0.9 ps and recovered in 20 ps. The ultrafast dynamics suggest that the magnetic degree of freedom dominates this magnetoelastic transition and ferromagnetic fluctuations might be universally relevant for this kind of compounds.

cond-mat.mtrl-sci↗

arXiv2020

Kernelized Multiview Subspace Analysis by Self-weighted Learning

With the popularity of multimedia technology, information is always represented or transmitted from multiple views. Most of the existing algorithms are graph-based ones to learn the complex structures within multiview data but overlooked the information within data representations. Furthermore, many existing works treat multiple views discriminatively by introducing some hyperparameters, which is undesirable in practice. To this end, abundant multiview based methods have been proposed for dimension reduction. However, there are still no research to leverage the existing work into a unified framework. To address this issue, in this paper, we propose a general framework for multiview data dimension reduction, named Kernelized Multiview Subspace Analysis (KMSA). It directly handles the multi-view feature representation in the kernel space, which provides a feasible channel for direct manipulations on multiview data with different dimensions. Meanwhile, compared with those graph-based methods, KMSA can fully exploit information from multiview data with nothing to lose. Furthermore, since different views have different influences on KMSA, we propose a self-weighted strategy to treat different views discriminatively according to their contributions. A co-regularized term is proposed to promote the mutual learning from multi-views. KMSA combines self-weighted learning with the co-regularized term to learn appropriate weights for all views. We also discuss the influence of the parameters in KMSA regarding the weights of multi-views. We evaluate our proposed framework on 6 multiview datasets for classification and image retrieval. The experimental results validate the advantages of our proposed method.

cs.LG↗

arXiv2020

Compressed DenseNet for Lightweight Character Recognition

Convolutional Recurrent Neural Network (CRNN) is a popular network for recognizing texts in images. Advances like the variant of CRNN, such as Dense Convolutional Network with Connectionist Temporal Classification, has reduced the running time of the network, but exposing the inner computation cost and weight size of the convolutional networks as a bottleneck. Specifically, the DenseNet based models utilize the dense blocks as the core module, but the inner features are combined in the form of concatenation in dense blocks. As such, the number of channels of combined features delivered as the input of the layers close to the output and the relevant computational cost grows rapidly with the dense blocks getting deeper. This will severely bring heavy computational cost and big weight size, which restrict the depth of dense blocks. In this paper, we propose a compressed convolution block called Lightweight Dense Block (LDB). To reduce the computing cost and weight size, we re-define and re-design the way of combining internal features of the dense blocks. LDB is a convolutional block similarly as dense block, but it can reduce the computation cost and weight size to (1/L, 2/L), compared with original ones, where L is the number of layers in blocks. Moreover, LDB can be used to replace the original dense block in any DenseNet based models. Based on the LDBs, we propose a Compressed DenseNet (CDenseNet) for the lightweight character recognition. Extensive experiments demonstrate that CDenseNet can effectively reduce the weight size while delivering the promising recognition results.

cs.CV↗

arXiv2020

LinSBFT: Linear-Communication One-Step BFT Protocol for Public Blockchains

This paper presents LinSBFT, a Byzantine Fault Tolerance (BFT) protocol with the capacity of processing over 2000 smart contract transactions per second in production. LinSBFT applies to a permissionless, public blockchain system, in which there is no public-key infrastructure, based on the classic PBFT with 4 improvements: (\romannumeral1) LinSBFT achieves $O(n)$ worst-case communication volume, in contract to PBFT's $O(n^4)$; (\romannumeral2) LinSBFT rotates the leader of protocol randomly to reduce the risk of denial-of-service attacks on leader; and (\romannumeral3) each run of LinSBFT finalizes one block, which is robust against participants that are honest in one run of the protocol, and dishonest in another, and the set of participants is dynamic, which is update periodically. (\romannumeral4) LinSBFT helps the delayed nodes to catch up via a synchronization mechanism to promise the liveness. Further, in the ordinary case, LinSBFT involves only a single round of voting instead of two in PBFT, which reduces both communication overhead and confirmation time, and employs the \emph{proof-of-stake} scheme to reward all participants. Extensive experiments using data obtained from the Ethereum demonstrate that LinSBFT consistently and significantly outperforms existing in-production BFT protocols for blockchains.

cs.DC↗

arXiv2020

Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks

The use of RGB-D information for salient object detection has been extensively explored in recent years. However, relatively few efforts have been put towards modeling salient object detection in real-world human activity scenes with RGBD. In this work, we fill the gap by making the following contributions to RGB-D salient object detection. (1) We carefully collect a new SIP (salient person) dataset, which consists of ~1K high-resolution images that cover diverse real-world scenes from various viewpoints, poses, occlusions, illuminations, and backgrounds. (2) We conduct a large-scale (and, so far, the most comprehensive) benchmark comparing contemporary methods, which has long been missing in the field and can serve as a baseline for future research. We systematically summarize 32 popular models and evaluate 18 parts of 32 models on seven datasets containing a total of about 97K images. (3) We propose a simple general architecture, called Deep Depth-Depurator Network (D3Net). It consists of a depth depurator unit (DDU) and a three-stream feature learning module (FLM), which performs low-quality depth map filtering and cross-modal feature learning respectively. These components form a nested structure and are elaborately designed to be learned jointly. D3Net exceeds the performance of any prior contenders across all five metrics under consideration, thus serving as a strong model to advance research in this field. We also demonstrate that D3Net can be used to efficiently extract salient object masks from real scenes, enabling effective background changing application with a speed of 65fps on a single GPU. All the saliency maps, our new SIP dataset, the D3Net model, and the evaluation tools are publicly available at https://github.com/DengPingFan/D3NetBenchmark.

cs.CV↗

arXiv2020

Convolutional Neural Network Training with Distributed K-FAC

Training neural networks with many processors can reduce time-to-solution; however, it is challenging to maintain convergence and efficiency at large scales. The Kronecker-factored Approximate Curvature (K-FAC) was recently proposed as an approximation of the Fisher Information Matrix that can be used in natural gradient optimizers. We investigate here a scalable K-FAC design and its applicability in convolutional neural network (CNN) training at scale. We study optimization techniques such as layer-wise distribution strategies, inverse-free second-order gradient evaluation, and dynamic K-FAC update decoupling to reduce training time while preserving convergence. We use residual neural networks (ResNet) applied to the CIFAR-10 and ImageNet-1k datasets to evaluate the correctness and scalability of our K-FAC gradient preconditioner. With ResNet-50 on the ImageNet-1k dataset, our distributed K-FAC implementation converges to the 75.9% MLPerf baseline in 18-25% less time than does the classic stochastic gradient descent (SGD) optimizer across scales on a GPU cluster.

cs.LG↗