arXiv subjects

Wen Shen

Publications and source records attributed to Wen Shen.

At least 55 records · Page 3Linked to original sources

arXiv2023

Defects of Convolutional Decoder Networks in Frequency Representation

In this paper, we prove the representation defects of a cascaded convolutional decoder network, considering the capacity of representing different frequency components of an input sample. We conduct the discrete Fourier transform on each channel of the feature map in an intermediate layer of the decoder network. Then, we extend the 2D circular convolution theorem to represent the forward and backward propagations through convolutional layers in the frequency domain. Based on this, we prove three defects in representing feature spectrums. First, we prove that the convolution operation, the zero-padding operation, and a set of other settings all make a convolutional decoder network more likely to weaken high-frequency components. Second, we prove that the upsampling operation generates a feature spectrum, in which strong signals repetitively appear at certain frequencies. Third, we prove that if the frequency components in the input sample and frequency components in the target output for regression have a small shift, then the decoder usually cannot be effectively learned.

cs.LG↗

arXiv2023

A Space-time Nonlocal Traffic Flow Model: Relaxation Representation and Local Limit

We propose and study a nonlocal conservation law modelling traffic flow in the existence of inter-vehicle communication. It is assumed that the nonlocal information travels at a finite speed and the model involves a space-time nonlocal integral of weighted traffic density. The well-posedness of the model is established under suitable conditions on the model parameters and by a suitably-defined initial condition. In a special case where the weight kernel in the nonlocal integral is an exponential function, the nonlocal model can be reformulated as a $2\times2$ hyperbolic system with relaxation. With the help of this relaxation representation, we show that the Lighthill-Whitham-Richards model is recovered in the equilibrium approximation limit.

math.AP↗

arXiv2023

Can the Inference Logic of Large Language Models be Disentangled into Symbolic Concepts?

In this paper, we explain the inference logic of large language models (LLMs) as a set of symbolic concepts. Many recent studies have discovered that traditional DNNs usually encode sparse symbolic concepts. However, because an LLM has much more parameters than traditional DNNs, whether the LLM also encodes sparse symbolic concepts is still an open problem. Therefore, in this paper, we propose to disentangle the inference score of LLMs for dialogue tasks into a small number of symbolic concepts. We verify that we can use those sparse concepts to well estimate all inference scores of the LLM on all arbitrarily masking states of the input sentence. We also evaluate the transferability of concepts encoded by an LLM and verify that symbolic concepts usually exhibit high transferability across similar input sentences. More crucially, those symbolic concepts can be used to explain the exact reasons accountable for the LLM's prediction errors.

cs.CL↗

arXiv2022

Batch Normalization Is Blind to the First and Second Derivatives of the Loss

In this paper, we prove the effects of the BN operation on the back-propagation of the first and second derivatives of the loss. When we do the Taylor series expansion of the loss function, we prove that the BN operation will block the influence of the first-order term and most influence of the second-order term of the loss. We also find that such a problem is caused by the standardization phase of the BN operation. Experimental results have verified our theoretical conclusions, and we have found that the BN operation significantly affects feature representations in specific tasks, where losses of different samples share similar analytic formulas.

cs.LG↗

arXiv2022

Why Adversarial Training of ReLU Networks Is Difficult?

This paper mathematically derives an analytic solution of the adversarial perturbation on a ReLU network, and theoretically explains the difficulty of adversarial training. Specifically, we formulate the dynamics of the adversarial perturbation generated by the multi-step attack, which shows that the adversarial perturbation tends to strengthen eigenvectors corresponding to a few top-ranked eigenvalues of the Hessian matrix of the loss w.r.t. the input. We also prove that adversarial training tends to strengthen the influence of unconfident input samples with large gradient norms in an exponential manner. Besides, we find that adversarial training strengthens the influence of the Hessian matrix of the loss w.r.t. network parameters, which makes the adversarial training more likely to oscillate along directions of a few samples, and boosts the difficulty of adversarial training. Crucially, our proofs provide a unified explanation for previous findings in understanding adversarial training.

cs.LG↗

arXiv2021

Interpreting Representation Quality of DNNs for 3D Point Cloud Processing

In this paper, we evaluate the quality of knowledge representations encoded in deep neural networks (DNNs) for 3D point cloud processing. We propose a method to disentangle the overall model vulnerability into the sensitivity to the rotation, the translation, the scale, and local 3D structures. Besides, we also propose metrics to evaluate the spatial smoothness of encoding 3D structures, and the representation complexity of the DNN. Based on such analysis, experiments expose representation problems with classic DNNs, and explain the utility of the adversarial training.

cs.CV↗

arXiv2021

Interpretable Compositional Convolutional Neural Networks

The reasonable definition of semantic interpretability presents the core challenge in explainable AI. This paper proposes a method to modify a traditional convolutional neural network (CNN) into an interpretable compositional CNN, in order to learn filters that encode meaningful visual patterns in intermediate convolutional layers. In a compositional CNN, each filter is supposed to consistently represent a specific compositional object part or image region with a clear meaning. The compositional CNN learns from image labels for classification without any annotations of parts or regions for supervision. Our method can be broadly applied to different types of CNNs. Experiments have demonstrated the effectiveness of our method.

cs.CV↗

arXiv2021

Verifiability and Predictability: Interpreting Utilities of Network Architectures for Point Cloud Processing

In this paper, we diagnose deep neural networks for 3D point cloud processing to explore utilities of different intermediate-layer network architectures. We propose a number of hypotheses on the effects of specific intermediate-layer network architectures on the representation capacity of DNNs. In order to prove the hypotheses, we design five metrics to diagnose various types of DNNs from the following perspectives, information discarding, information concentration, rotation robustness, adversarial robustness, and neighborhood inconsistency. We conduct comparative studies based on such metrics to verify the hypotheses. We further use the verified hypotheses to revise intermediate-layer architectures of existing DNNs and improve their utilities. Experiments demonstrate the effectiveness of our method.

cs.CV↗

arXiv2021

The Cauchy Problem for a non Strictly Hyperbolic $3\times3$ System of Conservation Laws Arising in Polymer Flooding

We study the Cauchy problem of a $3\times 3$ system of conservation laws modeling two--phase flow of polymer flooding in rough porous media with possibly discontinuous permeability function. The system loses strict hyperbolicity in some regions of the domain where the eigenvalues of different families coincide, and BV estimates are not available in general. For a suitable $2\times 2$ system, a singular change of variable introduced by Temple could be effective to control the total variation. An extension of this technique can be applied to a $3\times 3$ system only under strict hypotheses on the flux functions. In this paper, through an adapted front tracking algorithm we prove the existence of solutions for the Cauchy problem under mild assumptions on the flux function, using a compensated compactness argument.

math.AP↗

arXiv2020

Entropy Admissibility of the Limit Solution for a Nonlocal Model of Traffic Flow

math.AP↗

arXiv2020

3D-Rotation-Equivariant Quaternion Neural Networks

This paper proposes a set of rules to revise various neural networks for 3D point cloud processing to rotation-equivariant quaternion neural networks (REQNNs). We find that when a neural network uses quaternion features under certain conditions, the network feature naturally has the rotation-equivariance property. Rotation equivariance means that applying a specific rotation transformation to the input point cloud is equivalent to applying the same rotation transformation to all intermediate-layer quaternion features. Besides, the REQNN also ensures that the intermediate-layer features are invariant to the permutation of input points. Compared with the original neural network, the REQNN exhibits higher rotation robustness.

cs.CV↗

arXiv2020

A posteriori Error Estimates for Numerical Solutions to Hyperbolic Conservation Laws

The paper is concerned with a posteriori error bounds for a wide class of numerical schemes, for $n\times n$ hyperbolic conservation laws in one space dimension. These estimates are achieved by a "post-processing algorithm", checking that the numerical solution retains small total variation, and computing its oscillation on suitable subdomains. The results apply, in particular, to solutions obtained by the Godunov or the Lax-Friedrichs scheme, backward Euler approximations, and the method of periodic smoothing. Some numerical implementations are presented.

math.NA↗

arXiv2020

Automated Segmentation of Brain Gray Matter Nuclei on Quantitative Susceptibility Mapping Using Deep Convolutional Neural Network

Abnormal iron accumulation in the brain subcortical nuclei has been reported to be correlated to various neurodegenerative diseases, which can be measured through the magnetic susceptibility from the quantitative susceptibility mapping (QSM). To quantitively measure the magnetic susceptibility, the nuclei should be accurately segmented, which is a tedious task for clinicians. In this paper, we proposed a double-branch residual-structured U-Net (DB-ResUNet) based on 3D convolutional neural network (CNN) to automatically segment such brain gray matter nuclei. To better tradeoff between segmentation accuracy and the memory efficiency, the proposed DB-ResUNet fed image patches with high resolution and the patches with low resolution but larger field of view into the local and global branches, respectively. Experimental results revealed that by jointly using QSM and T$_\text{1}$ weighted imaging (T$_\text{1}$WI) as inputs, the proposed method was able to achieve better segmentation accuracy over its single-branch counterpart, as well as the conventional atlas-based method and the classical 3D-UNet structure. The susceptibility values and the volumes were also measured, which indicated that the measurements from the proposed DB-ResUNet are able to present high correlation with values from the manually annotated regions of interest.

eess.IV↗

arXiv2020

Spatial-Temporal Moving Target Defense: A Markov Stackelberg Game Model

Moving target defense has emerged as a critical paradigm of protecting a vulnerable system against persistent and stealthy attacks. To protect a system, a defender proactively changes the system configurations to limit the exposure of security vulnerabilities to potential attackers. In doing so, the defender creates asymmetric uncertainty and complexity for the attackers, making it much harder for them to compromise the system. In practice, the defender incurs a switching cost for each migration of the system configurations. The switching cost usually depends on both the current configuration and the following configuration. Besides, different system configurations typically require a different amount of time for an attacker to exploit and attack. Therefore, a defender must simultaneously decide both the optimal sequences of system configurations and the optimal timing for switching. In this paper, we propose a Markov Stackelberg Game framework to precisely characterize the defender's spatial and temporal decision-making in the face of advanced attackers. We introduce a relative value iteration algorithm that computes the defender's optimal moving target defense strategies. Empirical evaluation on real-world problems demonstrates the advantages of the Markov Stackelberg game model for spatial-temporal moving target defense.

cs.GT↗

arXiv2020

A Posteriori Error Estimates for Self-Similar Solutions to the Euler Equations

The main goal of this paper is to analyze a family of "simplest possible" initial data for which, as shown by numerical simulations, the incompressible Euler equations have multiple solutions. We take here a first step toward a rigorous validation of these numerical results. Namely, we consider the system of equations corresponding to a self-similar solution, restricted to a bounded domain with smooth boundary. Given an approximate solution obtained via a finite dimensional Galerkin method, we establish a posteriori error bounds on the distance between the numerical approximation and the exact solution having the same boundary data.

math.AP↗

arXiv2019

On Traffic Flow with Nonlocal Flux: a Relaxation Representation

We consider a conservation law model of traffic flow, where the velocity of each car depends on a weighted average of the traffic density $ρ$ ahead. The averaging kernel is of exponential type: $w_\varepsilon(s)=\varepsilon ^{-1} e^{-s/\varepsilon}$. By a transformation of coordinates, the problem can be reformulated as a $2\times 2$ hyperbolic system with relaxation. Uniform BV bounds on the solution are thus obtained, independent of the scaling parameter $\varepsilon $. Letting $\varepsilon\to 0$, the limit yields a weak solution to the corresponding conservation law $ρ_t + ( ρv(ρ))_x=0$. In the case where the velocity $v(ρ)= a-bρ$ is affine, using the Hardy-Littlewood rearrangement inequality we prove that the limit is the unique entropy-admissible solution to the scalar conservation law.

math.AP↗

arXiv2019

Stationary Wave Profiles for Nonlocal Particle Models of Traffic Flow on Rough Roads

We study a nonlocal particle model describing traffic flow on rough roads. In the model, each driver adjusts the speed of the car according to the condition over an interval in the front, leading to a system of nonlocal ODEs which we refer to as the FtLs (follow-the-leaders) model. Assuming that the road condition is discontinuous at the origin, we seek stationary wave profiles (see Definition 1.1) for the system of ODEs across this discontinuity. We derive a non-local delay differential equation with discontinuous coefficient, satisfied by the profiles, together with conditions on the asymptotic values as $x\to\pm\infty$. Results on existence, uniqueness, and local stability are proved, for all cases. We show that, depending on the case, there might exist a unique profile, infinitely many profiles, or no profiles. The stability result also depends on cases. Various numerical simulations are presented. Finally, we establish convergence of these profiles to those of a local particle model, as well as those of a nonlocal PDE model.

math.AP↗

arXiv2019

Multi-Winner Contests for Strategic Diffusion in Social Networks

Strategic diffusion encourages participants to take active roles in promoting stakeholders' agendas by rewarding successful referrals. As social media continues to transform the way people communicate, strategic diffusion has become a powerful tool for stakeholders to influence people's decisions or behaviors for desired objectives. Existing reward mechanisms for strategic diffusion are usually either vulnerable to false-name attacks or not individually rational for participants that have made successful referrals. Here, we introduce a novel multi-winner contests (MWC) mechanism for strategic diffusion in social networks. The MWC mechanism satisfies several desirable properties, including false-name-proofness, individual rationality, budget constraint, monotonicity, and subgraph constraint. Numerical experiments on four real-world social network datasets demonstrate that stakeholders can significantly boost participants' aggregated efforts with proper design of competitions. Our work sheds light on how to design manipulation-resistant mechanisms with appropriate contests.

cs.SI↗