arXiv subjects

Song Guo

Publications and source records attributed to Song Guo.

At least 181 records · Page 10Linked to original sources

arXiv2022

Rethinking Classifier and Adversarial Attack

Various defense models have been proposed to resist adversarial attack algorithms, but existing adversarial robustness evaluation methods always overestimate the adversarial robustness of these models (i.e., not approaching the lower bound of robustness). To solve this problem, this paper uses the proposed decouple space method to divide the classifier into two parts: non-linear and linear. Then, this paper defines the representation vector of the original example (and its space, i.e., the representation space) and uses the iterative optimization of Absolute Classification Boundaries Initialization (ACBI) to obtain a better attack starting point. Particularly, this paper applies ACBI to nearly 50 widely-used defense models (including 8 architectures). Experimental results show that ACBI achieves lower robust accuracy in all cases.

cs.LG↗

arXiv2022

Layer-wised Model Aggregation for Personalized Federated Learning

Personalized Federated Learning (pFL) not only can capture the common priors from broad range of distributed data, but also support customized models for heterogeneous clients. Researches over the past few years have applied the weighted aggregation manner to produce personalized models, where the weights are determined by calibrating the distance of the entire model parameters or loss values, and have yet to consider the layer-level impacts to the aggregation process, leading to lagged model convergence and inadequate personalization over non-IID datasets. In this paper, we propose a novel pFL training framework dubbed Layer-wised Personalized Federated learning (pFedLA) that can discern the importance of each layer from different clients, and thus is able to optimize the personalized model aggregation for clients with heterogeneous data. Specifically, we employ a dedicated hypernetwork per client on the server side, which is trained to identify the mutual contribution factors at layer granularity. Meanwhile, a parameterized mechanism is introduced to update the layer-wised aggregation weights to progressively exploit the inter-user similarity and realize accurate model personalization. Extensive experiments are conducted over different models and learning tasks, and we show that the proposed methods achieve significantly higher performance than state-of-the-art pFL methods.

cs.DC↗

arXiv2022

Personalized Federated Learning with Contextualized Generalization

The prevalent personalized federated learning (PFL) usually pursues a trade-off between personalization and generalization by maintaining a shared global model to guide the training process of local models. However, the sole global model may easily transfer deviated context knowledge to some local models when multiple latent contexts exist across the local datasets. In this paper, we propose a novel concept called contextualized generalization (CG) to provide each client with fine-grained context knowledge that can better fit the local data distributions and facilitate faster model convergence, based on which we properly design a framework of PFL, dubbed CGPFL. We conduct detailed theoretical analysis, in which the convergence guarantee is presented and $\mathcal{O}(\sqrt{K})$ speedup over most existing methods is granted. To quantitatively study the generalization-personalization trade-off, we introduce the 'generalization error' measure and prove that the proposed CGPFL can achieve a better trade-off than existing solutions. Moreover, our theoretical analysis further inspires a heuristic algorithm to find a near-optimal trade-off in CGPFL. Experimental results on multiple real-world datasets show that our approach surpasses the state-of-the-art methods on test accuracy by a significant margin.

cs.LG↗

arXiv2022

Efficient Attribute Unlearning: Towards Selective Removal of Input Attributes from Feature Representations

Recently, the enactment of privacy regulations has promoted the rise of the machine unlearning paradigm. Existing studies of machine unlearning mainly focus on sample-wise unlearning, such that a learnt model will not expose user's privacy at the sample level. Yet we argue that such ability of selective removal should also be presented at the attribute level, especially for the attributes irrelevant to the main task, e.g., whether a person recognized in a face recognition system wears glasses or the age range of that person. Through a comprehensive literature review, it is found that existing studies on attribute-related problems like fairness and de-biasing learning cannot address the above concerns properly. To bridge this gap, we propose a paradigm of selectively removing input attributes from feature representations which we name `attribute unlearning'. In this paradigm, certain attributes will be accurately captured and detached from the learned feature representations at the stage of training, according to their mutual information. The particular attributes will be progressively eliminated along with the training procedure towards convergence, while the rest of attributes related to the main task are preserved for achieving competitive model performance. Considering the computational complexity during the training process, we not only give a theoretically approximate training method, but also propose an acceleration scheme to speed up the training process. We validate our method by spanning several datasets and models and demonstrate that our design can preserve model fidelity and reach prevailing unlearning efficacy with high efficiency. The proposed unlearning paradigm builds a foundation for future machine unlearning system and will become an essential component of the latest privacy-related legislation.

cs.LG↗

arXiv2022

Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression

Traditional one-bit compressed stochastic gradient descent can not be directly employed in multi-hop all-reduce, a widely adopted distributed training paradigm in network-intensive high-performance computing systems such as public clouds. According to our theoretical findings, due to the cascading compression, the training process has considerable deterioration on the convergence performance. To overcome this limitation, we implement a sign-bit compression-based learning synchronization framework, Marsit. It prevents cascading compression via an elaborate bit-wise operation for unbiased sign aggregation and its specific global compensation mechanism for mitigating compression deviation. The proposed framework retains the same theoretical convergence rate as non-compression mechanisms. Experimental results demonstrate that Marsit reduces up to 35% training time while preserving the same accuracy as training without compression.

cs.LG↗

arXiv2022

Towards Unbiased Multi-label Zero-Shot Learning with Pyramid and Semantic Attention

Multi-label zero-shot learning extends conventional single-label zero-shot learning to a more realistic scenario that aims at recognizing multiple unseen labels of classes for each input sample. Existing works usually exploit attention mechanism to generate the correlation among different labels. However, most of them are usually biased on several major classes while neglect most of the minor classes with the same importance in input samples, and may thus result in overly diffused attention maps that cannot sufficiently cover minor classes. We argue that disregarding the connection between major and minor classes, i.e., correspond to the global and local information, respectively, is the cause of the problem. In this paper, we propose a novel framework of unbiased multi-label zero-shot learning, by considering various class-specific regions to calibrate the training process of the classifier. Specifically, Pyramid Feature Attention (PFA) is proposed to build the correlation between global and local information of samples to balance the presence of each class. Meanwhile, for the generated semantic representations of input samples, we propose Semantic Attention (SA) to strengthen the element-wise correlation among these vectors, which can encourage the coordinated representation of them. Extensive experiments on the large-scale multi-label zero-shot benchmarks NUS-WIDE and Open-Image demonstrate that the proposed method surpasses other representative methods by significant margins.

cs.CV↗

arXiv2022

A Coalition Formation Game Approach for Personalized Federated Learning

Facing the challenge of statistical diversity in client local data distribution, personalized federated learning (PFL) has become a growing research hotspot. Although the state-of-the-art methods with model similarity-based pairwise collaboration have achieved promising performance, they neglect the fact that model aggregation is essentially a collaboration process within the coalition, where the complex multiwise influences take place among clients. In this paper, we first apply Shapley value (SV) from coalition game theory into the PFL scenario. To measure the multiwise collaboration among a group of clients on the personalized learning performance, SV takes their marginal contribution to the final result as a metric. We propose a novel personalized algorithm: pFedSV, which can 1. identify each client's optimal collaborator coalition and 2. perform personalized model aggregation based on SV. Extensive experiments on various datasets (MNIST, Fashion-MNIST, and CIFAR-10) are conducted with different Non-IID data settings (Pathological and Dirichlet). The results show that pFedSV can achieve superior personalized accuracy for each client, compared to the state-of-the-art benchmarks.

cs.AI↗

arXiv2022

Federated Unlearning via Class-Discriminative Pruning

We explore the problem of selectively forgetting categories from trained CNN classification models in the federated learning (FL). Given that the data used for training cannot be accessed globally in FL, our insights probe deep into the internal influence of each channel. Through the visualization of feature maps activated by different channels, we observe that different channels have a varying contribution to different categories in image classification. Inspired by this, we propose a method for scrubbing the model clean of information about particular categories. The method does not require retraining from scratch, nor global access to the data used for training. Instead, we introduce the concept of Term Frequency Inverse Document Frequency (TF-IDF) to quantize the class discrimination of channels. Channels with high TF-IDF scores have more discrimination on the target categories and thus need to be pruned to unlearn. The channel pruning is followed by a fine-tuning process to recover the performance of the pruned model. Evaluated on CIFAR10 dataset, our method accelerates the speed of unlearning by 8.9x for the ResNet model, and 7.9x for the VGG model under no degradation in accuracy, compared to retraining from scratch. For CIFAR100 dataset, the speedups are 9.9x and 8.4x, respectively. We envision this work as a complementary block for FL towards compliance with legal and ethical criteria.

cs.CV↗

arXiv2021

Parameterized Knowledge Transfer for Personalized Federated Learning

In recent years, personalized federated learning (pFL) has attracted increasing attention for its potential in dealing with statistical heterogeneity among clients. However, the state-of-the-art pFL methods rely on model parameters aggregation at the server side, which require all models to have the same structure and size, and thus limits the application for more heterogeneous scenarios. To deal with such model constraints, we exploit the potentials of heterogeneous model settings and propose a novel training framework to employ personalized models for different clients. Specifically, we formulate the aggregation procedure in original pFL into a personalized group knowledge transfer training algorithm, namely, KT-pFL, which enables each client to maintain a personalized soft prediction at the server side to guide the others' local training. KT-pFL updates the personalized soft prediction of each client by a linear combination of all local soft predictions using a knowledge coefficient matrix, which can adaptively reinforce the collaboration among clients who own similar data distribution. Furthermore, to quantify the contributions of each client to others' personalized training, the knowledge coefficient matrix is parameterized so that it can be trained simultaneously with the models. The knowledge coefficient matrix and the model parameters are alternatively updated in each round following the gradient descent way. Extensive experiments on various datasets (EMNIST, Fashion\_MNIST, CIFAR-10) are conducted under different settings (heterogeneous models and data distributions). It is demonstrated that the proposed framework is the first federated learning paradigm that realizes personalized model training via parameterized group knowledge transfer while achieving significant performance gain comparing with state-of-the-art algorithms.

cs.LG↗

arXiv2021

SDN-based Resource Allocation in Edge and Cloud Computing Systems: An Evolutionary Stackelberg Differential Game Approach

Recently, the boosting growth of computation-heavy applications raises great challenges for the Fifth Generation (5G) and future wireless networks. As responding, the hybrid edge and cloud computing (ECC) system has been expected as a promising solution to handle the increasing computational applications with low-latency and on-demand services of computation offloading, which requires new computing resource sharing and access control technology paradigms. This work establishes a software-defined networking (SDN) based architecture for edge/cloud computing services in 5G heterogeneous networks (HetNets), which can support efficient and on-demand computing resource management to optimize resource utilization and satisfy the time-varying computational tasks uploaded by user devices. In addition, resulting from the information incompleteness, we design an evolutionary game based service selection for users, which can model the replicator dynamics of service subscription. Based on this dynamic access model, a Stackelberg differential game based cloud computing resource sharing mechanism is proposed to facilitate the resource trading between the cloud computing service provider (CCP) and different edge computing service providers (ECPs). Then we derive the optimal pricing and allocation strategies of cloud computing resource based on the replicator dynamics of users' service selection. These strategies can promise the maximum integral utilities to all computing service providers (CPs), meanwhile the user distribution can reach the evolutionary stable state at this Stackelberg equilibrium. Furthermore, simulation results validate the performance of the designed resource sharing mechanism, and reveal the convergence and equilibrium states of user selection, and computing resource pricing and allocation.

eess.SY↗

arXiv2021

Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training

Distributed asynchronous offline training has received widespread attention in recent years because of its high performance on large-scale data and complex models. As data are distributed from cloud-centric to edge nodes, a big challenge for distributed machine learning systems is how to handle native and natural non-independent and identically distributed (non-IID) data for training. Previous asynchronous training methods do not have a satisfying performance on non-IID data because it would result in that the training process fluctuates greatly which leads to an abnormal convergence. We propose a gradient scheduling algorithm with partly averaged gradients and global momentum (GSGM) for non-IID data distributed asynchronous training. Our key idea is to apply global momentum and local average to the biased gradient after scheduling, in order to make the training process steady. Experimental results show that for non-IID data training under the same experimental conditions, GSGM on popular optimization algorithms can achieve a 20% increase in training stability with a slight improvement in accuracy on Fashion-Mnist and CIFAR-10 datasets. Meanwhile, when expanding distributed scale on CIFAR-100 dataset that results in sparse data distribution, GSGM can perform a 37% improvement on training stability. Moreover, only GSGM can converge well when the number of computing nodes grows to 30, compared to the state-of-the-art distributed asynchronous algorithms. At the same time, GSGM is robust to different degrees of non-IID data.

cs.DC↗

arXiv2021

DPN: Detail-Preserving Network with High Resolution Representation for Efficient Segmentation of Retinal Vessels

Retinal vessels are important biomarkers for many ophthalmological and cardiovascular diseases. Hence, it is of great significance to develop automatic models for computer-aided diagnosis. Existing methods, such as U-Net follow the encoder-decoder pipeline, where detailed information is lost in the encoder in order to achieve a large field of view. Although spatial detailed information could be recovered partly in the decoder, while there is noise in the high-resolution feature maps of the encoder. And, we argue this encoder-decoder architecture is inefficient for vessel segmentation. In this paper, we present the detail-preserving network (DPN), which avoids the encoder-decoder pipeline. To preserve detailed information and learn structural information simultaneously, we designed the detail-preserving block (DP-Block). Further, we stacked eight DP-Blocks together to form the DPN. More importantly, there are no down-sampling operations among these blocks. Therefore, the DPN could maintain a high/full resolution during processing, avoiding the loss of detailed information. To illustrate the effectiveness of DPN, we conducted experiments over three public datasets. Experimental results show, compared to state-of-the-art methods, DPN shows competitive/better performance in terms of segmentation accuracy, segmentation speed, and model size. Specifically, 1) Our method achieves comparable segmentation performance on the DRIVE, CHASE_DB1, and HRF datasets. 2) The segmentation speed of DPN is over 20-160 times faster than other methods on the DRIVE dataset. 3) The number of parameters of DPN is1 around 120k, far less than all comparison methods.

eess.IV↗

arXiv2021

Decentralized Federated Learning for UAV Networks: Architecture, Challenges, and Opportunities

Unmanned aerial vehicles (UAVs), or say drones, are envisioned to support extensive applications in next-generation wireless networks in both civil and military fields. Empowering UAVs networks intelligence by artificial intelligence (AI) especially machine learning (ML) techniques is inevitable and appealing to enable the aforementioned applications. To solve the problems of traditional cloud-centric ML for UAV networks such as privacy concern, unacceptable latency, and resource burden, a distributed ML technique, \textit(i.e.), federated learning (FL), has been recently proposed to enable multiple UAVs to collaboratively train ML model without letting out raw data. However, almost all existing FL paradigms are still centralized, \textit{i.e.}, a central entity is in charge of ML model aggregation and fusion over the whole network, which could result in the issue of a single point of failure and are inappropriate to UAV networks with both unreliable nodes and links. Thus motivated, in this article, we propose a novel architecture called DFL-UN (\underline{D}ecentralized \underline{F}ederated \underline{L}earning for \underline{U}AV \underline{N}etworks), which enables FL within UAV networks without a central entity. We also conduct a preliminary simulation study to validate the feasibility and effectiveness of the DFL-UN architecture. Finally, we discuss the main challenges and potential research directions in the DFL-UN.

cs.NI↗

arXiv2021

Online Multiple Object Tracking with Cross-Task Synergy

Modern online multiple object tracking (MOT) methods usually focus on two directions to improve tracking performance. One is to predict new positions in an incoming frame based on tracking information from previous frames, and the other is to enhance data association by generating more discriminative identity embeddings. Some works combined both directions within one framework but handled them as two individual tasks, thus gaining little mutual benefits. In this paper, we propose a novel unified model with synergy between position prediction and embedding association. The two tasks are linked by temporal-aware target attention and distractor attention, as well as identity-aware memory aggregation model. Specifically, the attention modules can make the prediction focus more on targets and less on distractors, therefore more reliable embeddings can be extracted accordingly for association. On the other hand, such reliable embeddings can boost identity-awareness through memory aggregation, hence strengthen attention modules and suppress drifts. In this way, the synergy between position prediction and embedding association is achieved, which leads to strong robustness to occlusions. Extensive experiments demonstrate the superiority of our proposed model over a wide range of existing methods on MOTChallenge benchmarks. Our code and models are publicly available at https://github.com/songguocode/TADAM.

cs.CV↗

arXiv2021

Optimizing Video Caching at the Edge: A Hybrid Multi-Point Process Approach

It is always a challenging problem to deliver a huge volume of videos over the Internet. To meet the high bandwidth and stringent playback demand, one feasible solution is to cache video contents on edge servers based on predicted video popularity. Traditional caching algorithms (e.g., LRU, LFU) are too simple to capture the dynamics of video popularity, especially long-tailed videos. Recent learning-driven caching algorithms (e.g., DeepCache) show promising performance, however, such black-box approaches are lack of explainability and interpretability. Moreover, the parameter tuning requires a large number of historical records, which are difficult to obtain for videos with low popularity. In this paper, we optimize video caching at the edge using a white-box approach, which is highly efficient and also completely explainable. To accurately capture the evolution of video popularity, we develop a mathematical model called \emph{HRS} model, which is the combination of multiple point processes, including Hawkes' self-exciting, reactive and self-correcting processes. The key advantage of the HRS model is its explainability, and much less number of model parameters. In addition, all its model parameters can be learned automatically through maximizing the Log-likelihood function constructed by past video request events. Next, we further design an online HRS-based video caching algorithm. To verify its effectiveness, we conduct a series of experiments using real video traces collected from Tencent Video, one of the largest online video providers in China. Experiment results demonstrate that our proposed algorithm outperforms the state-of-the-art algorithms, with 12.3\% improvement on average in terms of cache hit rate under realistic settings.

cs.MM↗

arXiv2021

Gain without Pain: Offsetting DP-injected Nosies Stealthily in Cross-device Federated Learning

Federated Learning (FL) is an emerging paradigm through which decentralized devices can collaboratively train a common model. However, a serious concern is the leakage of privacy from exchanged gradient information between clients and the parameter server (PS) in FL. To protect gradient information, clients can adopt differential privacy (DP) to add additional noises and distort original gradients before they are uploaded to the PS. Nevertheless, the model accuracy will be significantly impaired by DP noises, making DP impracticable in real systems. In this work, we propose a novel Noise Information Secretly Sharing (NISS) algorithm to alleviate the disturbance of DP noises by sharing negated noises among clients. We theoretically prove that: 1) If clients are trustworthy, DP noises can be perfectly offset on the PS; 2) Clients can easily distort negated DP noises to protect themselves in case that other clients are not totally trustworthy, though the cost lowers model accuracy. NISS is particularly applicable for FL across multiple IoT (Internet of Things) systems, in which all IoT devices need to collaboratively train a model. To verify the effectiveness and the superiority of the NISS algorithm, we conduct experiments with the MNIST and CIFAR-10 datasets. The experiment results verify our analysis and demonstrate that NISS can improve model accuracy by 21% on average and obtain better privacy protection if clients are trustworthy.

cs.DC↗

arXiv2020

On the Convergence of Quantized Parallel Restarted SGD for Central Server Free Distributed Training

Communication is a crucial phase in the context of distributed training. Because parameter server (PS) frequently experiences network congestion, recent studies have found that training paradigms without a centralized server outperform the traditional server-based paradigms in terms of communication efficiency. However, with the increasing growth of model sizes, these server-free paradigms are also confronted with substantial communication overhead that seriously deteriorates the performance of distributed training. In this paper, we focus on communication efficiency of two serverless paradigms, i.e., Ring All-Reduce (RAR) and gossip, by proposing the Quantized Parallel Restarted Stochastic Gradient Descent (QPRSGD), an algorithm that allows multiple local SGD updates before a global synchronization, in synergy with the quantization to significantly reduce the communication overhead. We establish the bound of accumulative errors according to the synchronization mode and the network topology, which is essential to ensure the convergence property. Under both aggregation paradigms, the algorithm achieves the linear speedup property with respect to the number of local updates as well as the number of workers. Remarkably, the proposed algorithm achieves a convergence rate $O(1/\sqrt{NK^2M})$ under the gossip paradigm and outperforms all existing compression methods, where $N$ is the times of global synchronizations, and $K$ is the number of local updates, while $M$ is the number of nodes. An empirical study on various machine learning models demonstrates that the communication overhead is reduced by 90\%, and the convergence speed is boosted by up to 18.6 times, in a low bandwidth network, in comparison with Parallel SGD.

math.OC↗

arXiv2020

A Survey of State-of-the-Art on Blockchains: Theories, Modelings, and Tools

To draw a roadmap of current research activities of the blockchain community, we first conduct a brief overview of state-of-the-art blockchain surveys published in the recent 5 years. We found that those surveys are basically studying the blockchain-based applications, such as blockchain-assisted Internet of Things (IoT), business applications, security-enabled solutions, and many other applications in diverse fields. However, we think that a comprehensive survey towards the essentials of blockchains by exploiting the state-of-the-art theoretical modelings, analytic models, and useful experiment tools is still missing. To fill this gap, we perform a thorough survey by identifying and classifying the most recent high-quality research outputs that are closely related to the theoretical findings and essential mechanisms of blockchain systems and networks. Several promising open issues are also summarized finally for future research directions. We wish this survey can serve as a useful guideline for researchers, engineers, and educators about the cutting-edge development of blockchains in the perspectives of theories, modelings, and tools.

cs.DC↗