arXiv subjects

Hanlin Zhang

Publications and source records attributed to Hanlin Zhang.

47 records · Page 3Linked to original sources

arXiv2023

Iterative Graph Self-Distillation

Recently, there has been increasing interest in the challenge of how to discriminatively vectorize graphs. To address this, we propose a method called Iterative Graph Self-Distillation (IGSD) which learns graph-level representation in an unsupervised manner through instance discrimination using a self-supervised contrastive learning approach. IGSD involves a teacher-student distillation process that uses graph diffusion augmentations and constructs the teacher model using an exponential moving average of the student model. The intuition behind IGSD is to predict the teacher network representation of the graph pairs under different augmented views. As a natural extension, we also apply IGSD to semi-supervised scenarios by jointly regularizing the network with both supervised and self-supervised contrastive loss. Finally, we show that finetuning the IGSD-trained models with self-training can further improve the graph representation power. Empirically, we achieve significant and consistent performance gain on various graph datasets in both unsupervised and semi-supervised settings, which well validates the superiority of IGSD.

cs.LG↗

arXiv2022

A Closer Look at the Calibration of Differentially Private Learners

We systematically study the calibration of classifiers trained with differentially private stochastic gradient descent (DP-SGD) and observe miscalibration across a wide range of vision and language tasks. Our analysis identifies per-example gradient clipping in DP-SGD as a major cause of miscalibration, and we show that existing approaches for improving calibration with differential privacy only provide marginal improvements in calibration error while occasionally causing large degradations in accuracy. As a solution, we show that differentially private variants of post-processing calibration methods such as temperature scaling and Platt scaling are surprisingly effective and have negligible utility cost to the overall model. Across 7 tasks, temperature scaling and Platt scaling with DP-SGD result in an average 3.1-fold reduction in the in-domain expected calibration error and only incur at most a minor percent drop in accuracy.

cs.LG↗

arXiv2022

Towards Principled Disentanglement for Domain Generalization

A fundamental challenge for machine learning models is generalizing to out-of-distribution (OOD) data, in part due to spurious correlations. To tackle this challenge, we first formalize the OOD generalization problem as constrained optimization, called Disentanglement-constrained Domain Generalization (DDG). We relax this non-trivial constrained optimization problem to a tractable form with finite-dimensional parameterization and empirical approximation. Then a theoretical analysis of the extent to which the above transformations deviates from the original problem is provided. Based on the transformation, we propose a primal-dual algorithm for joint representation disentanglement and domain generalization. In contrast to traditional approaches based on domain adversarial training and domain labels, DDG jointly learns semantic and variation encoders for disentanglement, enabling flexible manipulation and augmentation on training data. DDG aims to learn intrinsic representations of semantic concepts that are invariant to nuisance factors and generalizable across domains. Comprehensive experiments on popular benchmarks show that DDG can achieve competitive OOD performance and uncover interpretable salient structures within data.

cs.LG↗

arXiv2022

Exploring Transformer Backbones for Heterogeneous Treatment Effect Estimation

Previous works on Treatment Effect Estimation (TEE) are not in widespread use because they are predominantly theoretical, where strong parametric assumptions are made but untractable for practical application. Recent work uses multilayer perceptron (MLP) for modeling casual relationships, however, MLPs lag far behind recent advances in ML methodology, which limits their applicability and generalizability. To extend beyond the single domain formulation and towards more realistic learning scenarios, we explore model design spaces beyond MLPs, i.e., transformer backbones, which provide flexibility where attention layers govern interactions among treatments and covariates to exploit structural similarities of potential outcomes for confounding control. Through careful model design, Transformers as Treatment Effect Estimators (TransTEE) is proposed. We show empirically that TransTEE can: (1) serve as a general purpose treatment effect estimator that significantly outperforms competitive baselines in a variety of challenging TEE problems (e.g., discrete, continuous, structured, or dosage-associated treatments) and is applicable to both when covariates are tabular and when they consist of structural data (e.g., texts, graphs); (2) yield multiple advantages: compatibility with propensity score modeling, parameter efficiency, robustness to continuous treatment value distribution shifts, explainable in covariate adjustment, and real-world utility in auditing pre-trained language models

cs.LG↗

arXiv2022

Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reduction

Deep neural networks are easily attacked by imperceptible perturbation. Presently, adversarial training (AT) is the most effective method to enhance the robustness of the model against adversarial examples. However, because adversarial training solved a min-max value problem, in comparison with natural training, the robustness and generalization are contradictory, i.e., the robustness improvement of the model will decrease the generalization of the model. To address this issue, in this paper, a new concept, namely confidence threshold (CT), is introduced and the reducing of the confidence threshold, known as confidence threshold reduction (CTR), is proven to improve both the generalization and robustness of the model. Specifically, to reduce the CT for natural training (i.e., for natural training with CTR), we propose a mask-guided divergence loss function (MDL) consisting of a cross-entropy loss term and an orthogonal term. The empirical and theoretical analysis demonstrates that the MDL loss improves the robustness and generalization of the model simultaneously for natural training. However, the model robustness improvement of natural training with CTR is not comparable to that of adversarial training. Therefore, for adversarial training, we propose a standard deviation loss function (STD), which minimizes the difference in the probabilities of the wrong categories, to reduce the CT by being integrated into the loss function of adversarial training. The empirical and theoretical analysis demonstrates that the STD based loss function can further improve the robustness of the adversarially trained model on basis of guaranteeing the changeless or slight improvement of the natural accuracy.

cs.LG↗

arXiv2022

Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Features

Machine learning has demonstrated remarkable prediction accuracy over i.i.d data, but the accuracy often drops when tested with data from another distribution. In this paper, we aim to offer another view of this problem in a perspective assuming the reason behind this accuracy drop is the reliance of models on the features that are not aligned well with how a data annotator considers similar across these two datasets. We refer to these features as misaligned features. We extend the conventional generalization error bound to a new one for this setup with the knowledge of how the misaligned features are associated with the label. Our analysis offers a set of techniques for this problem, and these techniques are naturally linked to many previous methods in robust machine learning literature. We also compared the empirical strength of these methods demonstrated the performance when these previous techniques are combined, with an implementation available at https://github.com/OoDBag/WR

cs.LG↗

arXiv2022

Towards Interpretable Natural Language Understanding with Explanations as Latent Variables

Recently generating natural language explanations has shown very promising results in not only offering interpretable explanations but also providing additional information and supervision for prediction. However, existing approaches usually require a large set of human annotated explanations for training while collecting a large set of explanations is not only time consuming but also expensive. In this paper, we develop a general framework for interpretable natural language understanding that requires only a small set of human annotated explanations for training. Our framework treats natural language explanations as latent variables that model the underlying reasoning process of a neural model. We develop a variational EM framework for optimization where an explanation generation module and an explanation-augmented prediction module are alternatively optimized and mutually enhance each other. Moreover, we further propose an explanation-based self-training method under this framework for semi-supervised learning. It alternates between assigning pseudo-labels to unlabeled data and generating new explanations to iteratively improve each other. Experiments on two natural language understanding tasks demonstrate that our framework can not only make effective predictions in both supervised and semi-supervised settings, but also generate good natural language explanation.

cs.CL↗

arXiv2022

Stochastic Neural Networks with Infinite Width are Deterministic

This work theoretically studies stochastic neural networks, a main type of neural network in use. We prove that as the width of an optimized stochastic neural network tends to infinity, its predictive variance on the training set decreases to zero. Our theory justifies the common intuition that adding stochasticity to the model can help regularize the model by introducing an averaging effect. Two common examples that our theory can be relevant to are neural networks with dropout and Bayesian latent variable models in a special limit. Our result thus helps better understand how stochasticity affects the learning of neural networks and potentially design better architectures for practical problems.

cs.LG↗

arXiv2021

Machine Learning on Cloud with Blockchain: A Secure, Verifiable and Fair Approach to Outsource the Linear Regression for Data Analysis

Linear Regression (LR) is a classical machine learning algorithm which has many applications in the cyber physical social systems (CPSS) to shape and simplify the way we live, work and communicate. This paper focuses on the data analysis for CPSS when the Linear Regression is applied. The training process of LR is time-consuming since it involves complex matrix operations, especially when it gets a large scale training dataset In the CPSS. Thus, how to enable devices to efficiently perform the training process of the Linear Regression is of significant importance. To address this issue, in this paper, we present a secure, verifiable and fair approach to outsource LR to an untrustworthy cloud-server. In the proposed scheme, computation inputs/outputs are obscured so that the privacy of sensitive information is protected against cloud-server. Meanwhile, computation result from cloud-server is verifiable. Also, fairness is guaranteed by the blockchain, which ensures that the cloud gets paid only if he correctly performed the outsourced workload. Based on the presented approach, we exploited the fair, secure outsourcing system on the Ethereum blockchain. We analysed our presented scheme on theoretical and experimental, all of which indicate that the presented scheme is valid, secure and efficient.

cs.CR↗

arXiv2021

Privacy-Preserving Cloud-Aided Broad Learning System

With the rapid development of artificial intelligence and the advent of the 5G era, deep learning has received extensive attention from researchers. Broad Learning System (BLS) is a new deep learning model proposed recently, which shows its effectiveness in many fields, such as image recognition and fault detection. However, the training process still requires vast computations, and therefore cannot be accomplished by some resource-constrained devices. To solve this problem, the resource-constrained device can outsource the BLS algorithm to cloud servers. Nevertheless, some security challenges also follow with the use of cloud computing, including the privacy of the data and the correctness of returned results. In this paper, we propose a secure, efficient, and verifiable outsourcing algorithm for BLS. This algorithm not only improves the efficiency of the algorithm on the client but also ensures that the clients sensitive information is not leaked to the cloud server. In addition, in our algorithm, the client can verify the correctness of returned results with a probability of almost 1. Finally, we analyze the security and efficiency of our algorithm in theory and prove our algorithms feasibility through experiments.

cs.CR↗

arXiv2021

Blockchain Aided Privacy-Preserving Outsourcing Algorithms of Bilinear Pairings for Internet of Things Devices

Bilinear pairing is a fundamental operation that is widely used in cryptographic algorithms (e.g., identity-based cryptographic algorithms) to secure IoT applications. Nonetheless, the time complexity of bilinear pairing is $O(n^3)$, making it a very time-consuming operation, especially for resource-constrained IoT devices. Secure outsourcing of bilinear pairing has been studied in recent years to enable computationally weak devices to securely outsource the bilinear pairing to untrustworthy cloud servers. However, the state-of-art algorithms often require to pre-compute and store some values, which results in storage burden for devices. In the Internet of Things, devices are generally with very limited storage capacity. Thus, the existing algorithms do not fit the IoT well. In this paper, we propose a secure outsourcing algorithm of bilinear pairings, which does not require pre-computations. In the proposed algorithm, the outsourcer side's efficiency is significantly improved compared with executing the original bilinear pairing operation. At the same time, the privacy of the input and output is ensured. Also, we apply the Ethereum blockchain in our outsourcing algorithm to enable fair payments, which ensures that the cloud server gets paid only when he correctly accomplished the outsourced work. The theoretical analysis and experimental results show that the proposed algorithm is efficient and secure.

cs.CR↗