arXiv subjects

Runyi Yu

Publications and source records attributed to Runyi Yu.

21 records · Page 2Linked to original sources

arXiv2022

$L_2$BN: Enhancing Batch Normalization by Equalizing the $L_2$ Norms of Features

In this paper, we analyze batch normalization from the perspective of discriminability and find the disadvantages ignored by previous studies: the difference in $l_2$ norms of sample features can hinder batch normalization from obtaining more distinguished inter-class features and more compact intra-class features. To address this issue, we propose a simple yet effective method to equalize the $l_2$ norms of sample features. Concretely, we $l_2$-normalize each sample feature before feeding them into batch normalization, and therefore the features are of the same magnitude. Since the proposed method combines the $l_2$ normalization and batch normalization, we name our method $L_2$BN. The $L_2$BN can strengthen the compactness of intra-class features and enlarge the discrepancy of inter-class features. The $L_2$BN is easy to implement and can exert its effect without any additional parameters or hyper-parameters. We evaluate the effectiveness of $L_2$BN through extensive experiments with various models on image classification and acoustic scene classification tasks. The results demonstrate that the $L_2$BN can boost the generalization ability of various neural network models and achieve considerable performance improvements.

cs.CV↗

arXiv2019

On the Global Optima of Kernelized Adversarial Representation Learning

Adversarial representation learning is a promising paradigm for obtaining data representations that are invariant to certain sensitive attributes while retaining the information necessary for predicting target attributes. Existing approaches solve this problem through iterative adversarial minimax optimization and lack theoretical guarantees. In this paper, we first study the "linear" form of this problem i.e., the setting where all the players are linear functions. We show that the resulting optimization problem is both non-convex and non-differentiable. We obtain an exact closed-form expression for its global optima through spectral learning and provide performance guarantees in terms of analytical bounds on the achievable utility and invariance. We then extend this solution and analysis to non-linear functions through kernel representation. Numerical experiments on UCI, Extended Yale B and CIFAR-100 datasets indicate that, (a) practically, our solution is ideal for "imparting" provable invariance to any biased pre-trained data representation, and (b) empirically, the trade-off between utility and invariance provided by our solution is comparable to iterative minimax optimization of existing deep neural network based approaches. Code is available at https://github.com/human-analysis/Kernel-ARL

cs.LG↗

arXiv2019

Constrained Sampling: Optimum Reconstruction in Subspace with Minimax Regret Constraint

This paper considers the problem of optimum reconstruction in generalized sampling-reconstruction processes (GSRPs). We propose constrained GSRP, a novel framework that minimizes the reconstruction error for inputs in a subspace, subject to a constraint on the maximum regret-error for any other signal in the entire signal space. This framework addresses the primary limitation of existing GSRPs (consistent, subspace and minimax regret), namely, the assumption that the \emph{a priori} subspace is either fully known or fully ignored. We formulate constrained GSRP as a constrained optimization problem, the solution to which turns out to be a convex combination of the subspace and the minimax regret samplings. Detailed theoretical analysis on the reconstruction error shows that constrained sampling achieves a reconstruction that is 1) (sub)optimal for signals in the input subspace, 2) robust for signals around the input subspace, and 3) reasonably bounded for any other signals with a simple choice of the constraint parameter. Experimental results on sampling-reconstruction of a Gaussian input and a speech signal demonstrate the effectiveness of the proposed scheme.

eess.SP↗