Searcharxiv⌕ Search

arXiv subjects

Jie Ding

Publications and source records attributed to Jie Ding.

At least 91 records · Page 5Linked to original sources
arXiv2022

Multimodal Controller for Generative Models

Class-conditional generative models are crucial tools for data generation from user-specified class labels. Existing approaches for class-conditional generative models require nontrivial modifications of backbone generative architectures to model conditional information fed into the model. This paper introduces a plug-and-play module named `multimodal controller' to generate multimodal data without introducing additional learning parameters. In the absence of the controllers, our model reduces to non-conditional generative models. We test the efficacy of multimodal controllers on CIFAR10, COIL100, and Omniglot benchmark datasets. We demonstrate that multimodal controlled generative models (including VAE, PixelCNN, Glow, and GAN) can generate class-conditional images of significantly better quality when compared with conditional generative models. Moreover, we show that multimodal controlled models can also create novel modalities of images.

cs.LG↗
arXiv2022

The Rate of Convergence of Variation-Constrained Deep Neural Networks

Multi-layer feedforward networks have been used to approximate a wide range of nonlinear functions. An important and fundamental problem is to understand the learnability of a network model through its statistical risk, or the expected prediction error on future data. To the best of our knowledge, the rate of convergence of neural networks shown by existing works is bounded by at most the order of $n^{-1/4}$ for a sample size of $n$. In this paper, we show that a class of variation-constrained neural networks, with arbitrary width, can achieve near-parametric rate $n^{-1/2+δ}$ for an arbitrarily small positive constant $δ$. It is equivalent to $n^{-1 +2δ}$ under the mean squared error. This rate is also observed by numerical experiments. The result indicates that the neural function space needed for approximating smooth functions may not be as large as what is often perceived. Our result also provides insight to the phenomena that deep neural networks do not easily suffer from overfitting when the number of neurons and learning parameters rapidly grow with $n$ or even surpass $n$. We also discuss the rate of convergence regarding other network parameters, including the input dimension, network layer, and coefficient norm.

cs.LG↗
arXiv2022

A Framework for Understanding Model Extraction Attack and Defense

The privacy of machine learning models has become a significant concern in many emerging Machine-Learning-as-a-Service applications, where prediction services based on well-trained models are offered to users via pay-per-query. The lack of a defense mechanism can impose a high risk on the privacy of the server's model since an adversary could efficiently steal the model by querying only a few `good' data points. The interplay between a server's defense and an adversary's attack inevitably leads to an arms race dilemma, as commonly seen in Adversarial Machine Learning. To study the fundamental tradeoffs between model utility from a benign user's view and privacy from an adversary's view, we develop new metrics to quantify such tradeoffs, analyze their theoretical properties, and develop an optimization problem to understand the optimal adversarial attack and defense strategies. The developed concepts and theory match the empirical findings on the `equilibrium' between privacy and utility. In terms of optimization, the key ingredient that enables our results is a unified representation of the attack-defense problem as a min-max bi-level problem. The developed results will be demonstrated by examples and experiments.

cs.LG↗
arXiv2022

Interval Privacy: A Framework for Privacy-Preserving Data Collection

The emerging public awareness and government regulations of data privacy motivate new paradigms of collecting and analyzing data that are transparent and acceptable to data owners. We present a new concept of privacy and corresponding data formats, mechanisms, and theories for privatizing data during data collection. The privacy, named Interval Privacy, enforces the raw data conditional distribution on the privatized data to be the same as its unconditional distribution over a nontrivial support set. Correspondingly, the proposed privacy mechanism will record each data value as a random interval (or, more generally, a range) containing it. The proposed interval privacy mechanisms can be easily deployed through survey-based data collection interfaces, e.g., by asking a respondent whether its data value is within a randomly generated range. Another unique feature of interval mechanisms is that they obfuscate the truth but do not perturb it. Using narrowed range to convey information is complementary to the popular paradigm of perturbing data. Also, the interval mechanisms can generate progressively refined information at the discretion of individuals, naturally leading to privacy-adaptive data collection. We develop different aspects of theory such as composition, robustness, distribution estimation, and regression learning from interval-valued data. Interval privacy provides a new perspective of human-centric data privacy where individuals have a perceptible, transparent, and simple way of sharing sensitive data.

cs.CR↗
arXiv2022

Self-Aware Personalized Federated Learning

In the context of personalized federated learning (FL), the critical challenge is to balance local model improvement and global model tuning when the personal and global objectives may not be exactly aligned. Inspired by Bayesian hierarchical models, we develop a self-aware personalized FL method where each client can automatically balance the training of its local personal model and the global model that implicitly contributes to other clients' training. Such a balance is derived from the inter-client and intra-client uncertainty quantification. A larger inter-client variation implies more personalization is needed. Correspondingly, our method uses uncertainty-driven local training steps and aggregation rule instead of conventional local fine-tuning and sample size-based aggregation. With experimental studies on synthetic data, Amazon Alexa audio data, and public datasets such as MNIST, FEMNIST, CIFAR10, and Sent140, we show that our proposed method can achieve significantly improved personalization performance compared with the existing counterparts.

cs.LG↗
arXiv2022

Convergence Analysis of Structure-Preserving Numerical Methods Based on Slotboom Transformation for the Poisson--Nernst--Planck Equations

The analysis of structure-preserving numerical methods for the Poisson--Nernst--Planck (PNP) system has attracted growing interests in recent years. In this work, we provide an optimal rate convergence analysis and error estimate for finite difference schemes based on the Slotboom reformulation. Different options of mobility average at the staggered mesh points are considered in the finite-difference spatial discretization, such as the harmonic mean, geometric mean, arithmetic mean, and entropic mean. A semi-implicit temporal discretization is applied, which in turn results in a non-constant coefficient, positive-definite linear system at each time step. A higher order asymptotic expansion is applied in the consistency analysis, and such a higher order consistency estimate is necessary to control the discrete maximum norm of the concentration variables. In convergence estimate, the harmonic mean for the mobility average, which turns out to bring lots of convenience in the theoretical analysis, is taken for simplicity, while other options of mobility average would also lead to the desired error estimate, with more technical details involved. As a result, an optimal rate convergence analysis on concentrations, electric potential, and ionic fluxes is derived, which is the first such results for the structure-preserving numerical schemes based on the Slotboom reformulation. It is remarked that the convergence analysis leads to a theoretical justification of the conditional energy dissipation analysis, which relies on the maximum norm bounds of the concentration and the gradient of the electric potential. Some numerical results are also presented to demonstrate the accuracy and structure-preserving performance of the associated schemes.

math.NA↗
arXiv2022

Targeted Cross-Validation

In many applications, we have access to the complete dataset but are only interested in the prediction of a particular region of predictor variables. A standard approach is to find the globally best modeling method from a set of candidate methods. However, it is perhaps rare in reality that one candidate method is uniformly better than the others. A natural approach for this scenario is to apply a weighted $L_2$ loss in performance assessment to reflect the region-specific interest. We propose a targeted cross-validation (TCV) to select models or procedures based on a general weighted $L_2$ loss. We show that the TCV is consistent in selecting the best performing candidate under the weighted $L_2$ loss. Experimental studies are used to demonstrate the use of TCV and its potential advantage over the global CV or the approach of using only local data for modeling a local region. Previous investigations on CV have relied on the condition that when the sample size is large enough, the ranking of two candidates stays the same. However, in many applications with the setup of changing data-generating processes or highly adaptive modeling methods, the relative performance of the methods is not static as the sample size varies. Even with a fixed data-generating process, it is possible that the ranking of two methods switches infinitely many times. In this work, we broaden the concept of the selection consistency by allowing the best candidate to switch as the sample size varies, and then establish the consistency of the TCV. This flexible framework can be applied to high-dimensional and complex machine learning scenarios where the relative performances of modeling procedures are dynamic.

stat.ML↗
arXiv2022

Is a Classification Procedure Good Enough? A Goodness-of-Fit Assessment Tool for Classification Learning

In recent years, many non-traditional classification methods, such as Random Forest, Boosting, and neural network, have been widely used in applications. Their performance is typically measured in terms of classification accuracy. While the classification error rate and the like are important, they do not address a fundamental question: Is the classification method underfitted? To our best knowledge, there is no existing method that can assess the goodness-of-fit of a general classification procedure. Indeed, the lack of a parametric assumption makes it challenging to construct proper tests. To overcome this difficulty, we propose a methodology called BAGofT that splits the data into a training set and a validation set. First, the classification procedure to assess is applied to the training set, which is also used to adaptively find a data grouping that reveals the most severe regions of underfitting. Then, based on this grouping, we calculate a test statistic by comparing the estimated success probabilities and the actual observed responses from the validation set. The data splitting guarantees that the size of the test is controlled under the null hypothesis, and the power of the test goes to one as the sample size increases under the alternative hypothesis. For testing parametric classification models, the BAGofT has a broader scope than the existing methods since it is not restricted to specific parametric models (e.g., logistic regression). Extensive simulation studies show the utility of the BAGofT when assessing general classification procedures and its strengths over some existing methods when testing parametric classification models.

stat.ML↗
arXiv2022

Federated Learning Challenges and Opportunities: An Outlook

Federated learning (FL) has been developed as a promising framework to leverage the resources of edge devices, enhance customers' privacy, comply with regulations, and reduce development costs. Although many methods and applications have been developed for FL, several critical challenges for practical FL systems remain unaddressed. This paper provides an outlook on FL development, categorized into five emerging directions of FL, namely algorithm foundation, personalization, hardware and security constraints, lifelong learning, and nonstandard data. Our unique perspectives are backed by practical observations from large-scale federated systems for edge devices.

cs.LG↗
arXiv2022

On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections

We propose a new structured pruning framework for compressing Deep Neural Networks (DNNs) with skip connections, based on measuring the statistical dependency of hidden layers and predicted outputs. The dependence measure defined by the energy statistics of hidden layers serves as a model-free measure of information between the feature maps and the output of the network. The estimated dependence measure is subsequently used to prune a collection of redundant and uninformative layers. Model-freeness of our measure guarantees that no parametric assumptions on the feature map distribution are required, making it computationally appealing for very high dimensional feature space in DNNs. Extensive numerical experiments on various architectures show the efficacy of the proposed pruning approach with competitive performance to state-of-the-art methods.

cs.LG↗
arXiv2022

Asymptotic Critical Transmission Radii in Wireless Networks over a Convex Region

Critical transmission ranges (or radii) in wireless ad-hoc and sensor networks have been extensively investigated for various performance metrics such as connectivity, coverage, power assignment and energy consumption. However, the regions on which the networks are distributed are typically either squares or disks in existing works, which seriously limits the usage in real-life applications. In this article, we consider a convex region (i.e., a generalisation of squares and disks) on which wireless nodes are uniformly distributed. We have investigated two types of critical transmission radii, defined in terms of k-connectivity and the minimum vertex degree, respectively, and have also established their precise asymptotic distributions. These make the previous results obtained under the circumstance of squares or disks special cases of this work. More importantly, our results reveal how the region shape impacts on the critical transmission ranges: it is the length of the boundary of the (fixed-area) region that completely determines the transmission ranges. Furthermore, by isodiametric inequality, the smallest critical transmission ranges are achieved when regions are disks only.

cs.IT↗
arXiv2021

SPIDER: Searching Personalized Neural Architecture for Federated Learning

Federated learning (FL) is an efficient learning framework that assists distributed machine learning when data cannot be shared with a centralized server due to privacy and regulatory restrictions. Recent advancements in FL use predefined architecture-based learning for all the clients. However, given that clients' data are invisible to the server and data distributions are non-identical across clients, a predefined architecture discovered in a centralized setting may not be an optimal solution for all the clients in FL. Motivated by this challenge, in this work, we introduce SPIDER, an algorithmic framework that aims to Search Personalized neural architecture for federated learning. SPIDER is designed based on two unique features: (1) alternately optimizing one architecture-homogeneous global model (Supernet) in a generic FL manner and one architecture-heterogeneous local model that is connected to the global model by weight sharing-based regularization (2) achieving architecture-heterogeneous local model by a novel neural architecture search (NAS) method that can select optimal subnet progressively using operation-level perturbation on the accuracy value as the criterion. Experimental results demonstrate that SPIDER outperforms other state-of-the-art personalization methods, and the searched personalized architectures are more inference efficient.

cs.LG↗
arXiv2021

Intrinsic signal optoretinography of dark adaptation kinetics

Delayed dark adaptation due to impaired rod photoreceptor homeostasis has been reported as the earliest symptom of eye diseases such as age-related macular degeneration, diabetic retinopathy, and retinitis pigmentosa. Objective measurement of dark adaptation can facilitate early diagnosis to enable prompt intervention to prevent vision losses. However, there is a lack of noninvasive methods capable of spatiotemporal monitoring of photoreceptor changes during dark adaptation. Here we demonstrate functional optical coherence tomography (OCT) for in vivo intrinsic signal optoretinography (ORG) of dark adaptation kinetics in the C57BL/6J mouse retina. Functional OCT revealed a shortening of the outer retina, a morphological change in the cone and rod photoreceptor interdigitation zone, and a reduction in intrinsic signal amplitude at the photoreceptor inner segment ellipsoid. A strong positive correlation between morphophysiological activities was also confirmed. Functional OCT of dark adaptation kinetics promises a method for rapid ORG assessment of physiological integrity of retinal photoreceptors.

physics.bio-ph↗
arXiv2021

HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients

Federated Learning (FL) is a method of training machine learning models on private data distributed over a large number of possibly heterogeneous clients such as mobile phones and IoT devices. In this work, we propose a new federated learning framework named HeteroFL to address heterogeneous clients equipped with very different computation and communication capabilities. Our solution can enable the training of heterogeneous local models with varying computation complexities and still produce a single global inference model. For the first time, our method challenges the underlying assumption of existing work that local models have to share the same architecture as the global model. We demonstrate several strategies to enhance FL training and conduct extensive empirical evaluations, including five computation complexity levels of three model architecture on three datasets. We show that adaptively distributing subnetworks according to clients' capabilities is both computation and communication efficient.

cs.LG↗
arXiv2021

Model Linkage Selection for Cooperative Learning

We consider a distributed learning setting where each agent/learner holds a specific parametric model and data source. The goal is to integrate information across a set of learners to enhance the prediction accuracy of a given learner. A natural way to integrate information is to build a joint model across a group of learners that shares common parameters of interest. However, the underlying parameter sharing patterns across a set of learners may not be a priori known. Misspecifying the parameter sharing patterns or the parametric model for each learner often yields a biased estimation and degrades the prediction accuracy. We propose a general method to integrate information across a set of learners that is robust against misspecifications of both models and parameter sharing patterns. The main crux is to sequentially incorporate additional learners that can enhance the prediction accuracy of an existing joint model based on user-specified parameter sharing patterns across a set of learners. Theoretically, we show that the proposed method can data-adaptively select the most suitable way of parameter sharing and thus enhance the predictive performance of any particular learner of interest. Extensive numerical studies show the promising performance of the proposed method.

stat.ME↗
arXiv2021

Subset Privacy: Draw from an Obfuscated Urn

With the rapidly increasing ability to collect and analyze personal data, data privacy becomes an emerging concern. In this work, we develop a new statistical notion of local privacy to protect each categorical data that will be collected by untrusted entities. The proposed solution, named subset privacy, privatizes the original data value by replacing it with a random subset containing that value. We develop methods for the estimation of distribution functions and independence testing from subset-private data with theoretical guarantees. We also study different mechanisms to realize the subset privacy and evaluation metrics to quantify the amount of privacy in practice. Experimental results on both simulated and real-world datasets demonstrate the encouraging performance of the developed concepts and methods.

cs.CR↗
arXiv2021

Route to High-Performance Micro-solid Oxide Fuel Cells on Metallic Substrates

Micro-solid oxide fuel cells based on thin films have strong potential for use in portable power devices. However, devices based on silicon substrates typically involve thin-film metallic electrodes which are unstable at high temperatures. Devices based on bulk metal substrates overcome these limitations, though performance is hindered by the challenge of growing state-of-the-art epitaxial materials on metals. Here, we demonstrate for the first time the growth of epitaxial cathode materials on metal substrates (stainless steel) commercially supplied with epitaxial electrolyte layers (1.5 {um (Y2O3)0.15(ZrO2)0.85 (YSZ) + 50 nm CeO2). We create epitaxial mesoporous cathodes of (La0.60Sr0.40)0.95Co0.20Fe0.80O3 (LSCF) on the substrate by growing LSCF/MgO vertically aligned nanocomposite films by pulsed laser deposition, followed by selectively etching out the MgO. To enable valid comparison with the literature, the cathodes are also grown on single-crystal substrates, confirming state-of-the-art performance with an area specific resistance of 100ohmegacm2 at 500dC and activation energy down to 0.97 eV. The work marks an important step toward the commercialization of high-performance micro-solid oxide fuel cells for portable power applications.

physics.app-ph↗