arXiv subjects

Wang Lu

Publications and source records attributed to Wang Lu.

33 records · Page 2Linked to original sources

arXiv2023

MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare

Federated learning has attracted increasing attention to building models without accessing the raw user data, especially in healthcare. In real applications, different federations can seldom work together due to possible reasons such as data heterogeneity and distrust/inexistence of the central server. In this paper, we propose a novel framework called MetaFed to facilitate trustworthy FL between different federations. MetaFed obtains a personalized model for each federation without a central server via the proposed Cyclic Knowledge Distillation. Specifically, MetaFed treats each federation as a meta distribution and aggregates knowledge of each federation in a cyclic manner. The training is split into two parts: common knowledge accumulation and personalization. Comprehensive experiments on three benchmarks demonstrate that MetaFed without a server achieves better accuracy compared to state-of-the-art methods (e.g., 10%+ accuracy improvement compared to the baseline for PAMAP2) with fewer communication costs.

cs.LG↗

arXiv2023

FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning

Federated learning (FL) has emerged as a new paradigm for privacy-preserving computation in recent years. Unfortunately, FL faces two critical challenges that hinder its actual performance: data distribution heterogeneity and high resource costs brought by large foundation models. Specifically, the non-IID data in different clients make existing FL algorithms hard to converge while the high resource costs, including computational and communication costs that increase the deployment difficulty in real-world scenarios. In this paper, we propose an effective yet simple method, named FedCLIP, to achieve fast generalization and personalization for CLIP in federated learning. Concretely, we design an attention-based adapter for the large model, CLIP, and the rest operations merely depend on adapters. Lightweight adapters can make the most use of pretrained model information and ensure models be adaptive for clients in specific tasks. Simultaneously, small-scale operations can mitigate the computational burden and communication burden caused by large models. Extensive experiments are conducted on three datasets with distribution shifts. Qualitative and quantitative results demonstrate that FedCLIP significantly outperforms other baselines (9% overall improvements on PACS) and effectively reduces computational and communication costs (283x faster than FedAVG). Our code will be available at: https://github.com/microsoft/PersonalizedFL.

cs.LG↗

arXiv2023

Generalizable Low-Resource Activity Recognition with Diverse and Discriminative Representation Learning

Human activity recognition (HAR) is a time series classification task that focuses on identifying the motion patterns from human sensor readings. Adequate data is essential but a major bottleneck for training a generalizable HAR model, which assists customization and optimization of online web applications. However, it is costly in time and economy to collect large-scale labeled data in reality, i.e., the low-resource challenge. Meanwhile, data collected from different persons have distribution shifts due to different living habits, body shapes, age groups, etc. The low-resource and distribution shift challenges are detrimental to HAR when applying the trained model to new unseen subjects. In this paper, we propose a novel approach called Diverse and Discriminative representation Learning (DDLearn) for generalizable low-resource HAR. DDLearn simultaneously considers diversity and discrimination learning. With the constructed self-supervised learning task, DDLearn enlarges the data diversity and explores the latent activity properties. Then, we propose a diversity preservation module to preserve the diversity of learned features by enlarging the distribution divergence between the original and augmented domains. Meanwhile, DDLearn also enhances semantic discrimination by learning discriminative representations with supervised contrastive learning. Extensive experiments on three public HAR datasets demonstrate that our method significantly outperforms state-of-art methods by an average accuracy improvement of 9.5% under the low-resource distribution shift scenarios, while being a generic, explainable, and flexible framework. Code is available at: https://github.com/microsoft/robustlearn.

cs.CV↗

arXiv2023

Out-of-Distribution Representation Learning for Time Series Classification

Time series classification is an important problem in real world. Due to its non-stationary property that the distribution changes over time, it remains challenging to build models for generalization to unseen distributions. In this paper, we propose to view the time series classification problem from the distribution perspective. We argue that the temporal complexity attributes to the unknown latent distributions within. To this end, we propose DIVERSIFY to learn generalized representations for time series classification. DIVERSIFY takes an iterative process: it first obtains the worst-case distribution scenario via adversarial training, then matches the distributions of the obtained sub-domains. We also present some theoretical insights. We conduct experiments on gesture recognition, speech commands recognition, wearable stress and affect detection, and sensor-based human activity recognition with a total of seven datasets in different settings. Results demonstrate that DIVERSIFY significantly outperforms other baselines and effectively characterizes the latent distributions by qualitative and quantitative analysis. Code is available at: https://github.com/microsoft/robustlearn.

cs.LG↗

arXiv2022

Domain-invariant Feature Exploration for Domain Generalization

Deep learning has achieved great success in the past few years. However, the performance of deep learning is likely to impede in face of non-IID situations. Domain generalization (DG) enables a model to generalize to an unseen test distribution, i.e., to learn domain-invariant representations. In this paper, we argue that domain-invariant features should be originating from both internal and mutual sides. Internal invariance means that the features can be learned with a single domain and the features capture intrinsic semantics of data, i.e., the property within a domain, which is agnostic to other domains. Mutual invariance means that the features can be learned with multiple domains (cross-domain) and the features contain common information, i.e., the transferable features w.r.t. other domains. We then propose DIFEX for Domain-Invariant Feature EXploration. DIFEX employs a knowledge distillation framework to capture the high-level Fourier phase as the internally-invariant features and learn cross-domain correlation alignment as the mutually-invariant features. We further design an exploration loss to increase the feature diversity for better generalization. Extensive experiments on both time-series and visual benchmarks demonstrate that the proposed DIFEX achieves state-of-the-art performance.

cs.LG↗

arXiv2022

Domain Generalization for Activity Recognition via Adaptive Feature Fusion

Human activity recognition requires the efforts to build a generalizable model using the training datasets with the hope to achieve good performance in test datasets. However, in real applications, the training and testing datasets may have totally different distributions due to various reasons such as different body shapes, acting styles, and habits, damaging the model's generalization performance. While such a distribution gap can be reduced by existing domain adaptation approaches, they typically assume that the test data can be accessed in the training stage, which is not realistic. In this paper, we consider a more practical and challenging scenario: domain-generalized activity recognition (DGAR) where the test dataset \emph{cannot} be accessed during training. To this end, we propose \emph{Adaptive Feature Fusion for Activity Recognition~(AFFAR)}, a domain generalization approach that learns to fuse the domain-invariant and domain-specific representations to improve the model's generalization performance. AFFAR takes the best of both worlds where domain-invariant representations enhance the transferability across domains and domain-specific representations leverage the model discrimination power from each domain. Extensive experiments on three public HAR datasets show its effectiveness. Furthermore, we apply AFFAR to a real application, i.e., the diagnosis of Children's Attention Deficit Hyperactivity Disorder~(ADHD), which also demonstrates the superiority of our approach.

cs.CV↗

arXiv2022

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

It is expensive and time-consuming to collect sufficient labeled data to build human activity recognition (HAR) models. Training on existing data often makes the model biased towards the distribution of the training data, thus the model might perform terribly on test data with different distributions. Although existing efforts on transfer learning and domain adaptation try to solve the above problem, they still need access to unlabeled data on the target domain, which may not be possible in real scenarios. Few works pay attention to training a model that can generalize well to unseen target domains for HAR. In this paper, we propose a novel method called Semantic-Discriminative Mixup (SDMix) for generalizable cross-domain HAR. Firstly, we introduce semantic-aware Mixup that considers the activity semantic ranges to overcome the semantic inconsistency brought by domain differences. Secondly, we introduce the large margin loss to enhance the discrimination of Mixup to prevent misclassification brought by noisy virtual labels. Comprehensive generalization experiments on five public datasets demonstrate that our SDMix substantially outperforms the state-of-the-art approaches with 6% average accuracy improvement on cross-person, cross-dataset, and cross-position HAR.

cs.AI↗

arXiv2022

Generalizing to Unseen Domains: A Survey on Domain Generalization

Machine learning systems generally assume that the training and testing distributions are the same. To this end, a key requirement is to develop models that can generalize to unseen distributions. Domain generalization (DG), i.e., out-of-distribution generalization, has attracted increasing interests in recent years. Domain generalization deals with a challenging setting where one or several different but related domain(s) are given, and the goal is to learn a model that can generalize to an unseen test domain. Great progress has been made in the area of domain generalization for years. This paper presents the first review of recent advances in this area. First, we provide a formal definition of domain generalization and discuss several related fields. We then thoroughly review the theories related to domain generalization and carefully analyze the theory behind generalization. We categorize recent algorithms into three classes: data manipulation, representation learning, and learning strategy, and present several popular algorithms in detail for each category. Third, we introduce the commonly used datasets, applications, and our open-sourced codebase for fair evaluation. Finally, we summarize existing literature and present some potential research topics for the future.

cs.LG↗

arXiv2022

Personalized Federated Learning with Adaptive Batchnorm for Healthcare

There is a growing interest in applying machine learning techniques to healthcare. Recently, federated learning (FL) is gaining popularity since it allows researchers to train powerful models without compromising data privacy and security. However, the performance of existing FL approaches often deteriorates when encountering non-iid situations where there exist distribution gaps among clients, and few previous efforts focus on personalization in healthcare. In this article, we propose FedAP to tackle domain shifts and then obtain personalized models for local clients. FedAP learns the similarity between clients based on the statistics of the batch normalization layers while preserving the specificity of each client with different local batch normalization. Comprehensive experiments on five healthcare benchmarks demonstrate that FedAP achieves better accuracy compared to state-of-the-art methods (e.g., 10% accuracy improvement for PAMAP2) with faster convergence speed.

cs.LG↗

arXiv2021

Cross-domain Activity Recognition via Substructural Optimal Transport

It is expensive and time-consuming to collect sufficient labeled data for human activity recognition (HAR). Domain adaptation is a promising approach for cross-domain activity recognition. Existing methods mainly focus on adapting cross-domain representations via domain-level, class-level, or sample-level distribution matching. However, they might fail to capture the fine-grained locality information in activity data. The domain- and class-level matching are too coarse that may result in under-adaptation, while sample-level matching may be affected by the noise seriously and eventually cause over-adaptation. In this paper, we propose substructure-level matching for domain adaptation (SSDA) to better utilize the locality information of activity data for accurate and efficient knowledge transfer. Based on SSDA, we propose an optimal transport-based implementation, Substructural Optimal Transport (SOT), for cross-domain HAR. We obtain the substructures of activities via clustering methods and seeks the coupling of the weighted substructures between different domains. We conduct comprehensive experiments on four public activity recognition datasets (i.e. UCI-DSADS, UCI-HAR, USC-HAD, PAMAP2), which demonstrates that SOT significantly outperforms other state-of-the-art methods w.r.t classification accuracy (9%+ improvement). In addition, our mehtod is 5x faster than traditional OT-based DA methods with the same hyper-parameters.

eess.SP↗

arXiv2021

FedHealth 2: Weighted Federated Transfer Learning via Batch Normalization for Personalized Healthcare

The success of machine learning applications often needs a large quantity of data. Recently, federated learning (FL) is attracting increasing attention due to the demand for data privacy and security, especially in the medical field. However, the performance of existing FL approaches often deteriorates when there exist domain shifts among clients, and few previous works focus on personalization in healthcare. In this article, we propose FedHealth 2, an extension of FedHealth \cite{chen2020fedhealth} to tackle domain shifts and get personalized models for local clients. FedHealth 2 obtains the client similarities via a pretrained model, and then it averages all weighted models with preserving local batch normalization. Wearable activity recognition and COVID-19 auxiliary diagnosis experiments have evaluated that FedHealth 2 can achieve better accuracy (10%+ improvement for activity recognition) and personalized healthcare without compromising privacy and security.

cs.LG↗

arXiv2021

Searching Ternary Oxides and Chalcogenides as Positive Electrodes for Calcium Batteries

The identification of alternatives to the Lithium-ion battery architecture remains a crucial priority in the diversification of energy storage technologies. Accompanied by the low reduction potential of $\mathrm{Ca^{2+}/Ca}$, -2.87 V vs. SHE, metal-anode-based rechargeable Calcium (Ca) batteries appear competitive in terms of energy densities. However, the development of Ca-batteries lacks high-energy density intercalation cathode materials. Using first-principles methodologies, we screen a large chemical space for potential Ca-based cathode chemistries, with composition of $\mathrm{Ca_iTM_jZ_k}$, where TM is a 1$^{st}$ or 2$^{nd}$ row transition metal and $\mathrm{Z}$ is oxygen, sulfur, selenium or tellurium. 10 materials are selected and their Ca intercalation properties are investigated. We identify two previously unreported promising electrode compositions: the post-spinel $\mathrm{CaV_2O_4}$ and the layered $\mathrm{CaNb_2O_4}$, with Ca migration barriers of $\sim$654 meV and $\sim$785 meV, respectively. Finally, we analyse the geometrical features of the Ca migration pathways across the 10 materials studied and provide an updated set of design rules for the identification of good ionic conductors, especially with large mobile cations.

cond-mat.mtrl-sci↗

arXiv2020

Strain-Rate-Dependent Deformation Behavior of Ti29Zr24Nb23Hf24 High Entropy Alloys at Elevated and Room Temperature

We investigate the strain-rate-dependent mechanical behavior and deformation mechanisms of a refractory high entropy alloy, Ti29Zr24Nb23Hf24 (at.%), with a single-phase body-centered cubic (BCC) structure. High-temperature compression tests were conducted at temperatures from 700 to 1100°C at strain rates ranging from 10-3 to 10 s-1. A sudden stress drop after yield point was observed at higher temperatures and lower strain rates with the Zener-Holloman parameter, lnZ, in the range of 17.2-20.7. Such a softening behavior can be related to the interaction of dislocations with short-range clustering. However, at higher strain rates or lower temperatures (lnZ>25.0), kink bands were activated being responsible for the continuous strengthening of the alloy in competition with the softening mechanism. Systematic TEM investigations reveal that dislocation walls formed along {110} planes and dislocation loops were observed at a low strain of 6% at a high strain rate of 1 s-1 and 800°C. Kink band induced dynamic recrystallization is evident upon further straining. On the other hand, at low strain rate of 10-3 s-1 and 800°C, discontinuous recrystallization mechanisms become dominant with arrays of dislocations forming in front of the bulged boundaries of parent grains. These sub-grain boundaries eventually turn into high-angle grain boundaries. We also investigate the deformation mechanism of the alloy under extremely high strain rate (103 s-1) at room temperature. The specimen exhibits extensive kink bands with arrays of dislocation walls. As further strained, multiple slip systems can be activated and the interaction of dislocation walls plays a vital role in the strain hardening of the alloy.

cond-mat.mtrl-sci↗

arXiv2020

Statistical properties of radio flux densities of solar flares

Short timescale flux variations are closely related to the energy release process of magnetic reconnection during solar flares. Radio light curves at 1, 2, 3.75, 9.4, and 17 GHz of 209 flares observed by the Nobeyama Radio Polarimeter from 2000 to 2010 are analyzed with a running smooth technique. We find that the impulsive component (with a variation timescale shorter than 1 second) of 1 GHz emission of most flares peaks at a few tens of solar flux unit and lasts for about 1 minute and the impulsive component of 2 GHz emission lasts a shorter period and peaks at a lower flux level, while at the three high frequency channels the occurrence frequency of flares increases with the decrease of the flux density up to the noise level of the corresponding background. The gradual components of these emissions, however, have similar duration and peak flux density distributions. We also derive the power spectrum on different timescales and a normalized wavelet analysis is used to confirm features on short timescales. At a time resolution of 0.1 second, more than $\sim$ 60$\%$ of these radio light curves show significant flux variation on 1 second or shorter time scales. This fraction increases with the decrease of frequency and reaches $\sim$ 100$\%$ at 1 GHz, implying that short timescale processes are universal in solar flares. We also study the correlation between the impulsive radio flux densities and soft X-ray fluxes obtained with the GOES satellites and find that more than 65$\%$ of the flares with an impulsive component have their impulsive radio emission reach a peak value ahead of the soft X-ray fluxes and this fraction increases with the radio frequency.

astro-ph.SR↗

arXiv2014

PET image reconstruction with system matrix containing point spread function derived from single photon incidence response

In positron emission tomography (PET) imaging, statistical iterative reconstruction (IR) techniques appear particularly promising since they can provide accurate system model. The system model matrix which describes the relationship between image space and projection space is important to the image quality. It contains some factors such as geometrical component and blurring component. The blurring component is usually described by point spread function (PSF). A PSF matrix derived from the single photon incidence response function is studied. And then an IR method based on the system matrix containing the PSF is developed. More specifically, the gamma photon incidence on a crystal array is simulated by Monte Carlo (MC) simulation, and then the single photon incidence response functions are calculated. Subsequently, the single photon incidence response functions is used to compute the coincidence blurring factor according to the physical process of PET coincidence detection. Through weighting the ordinary system matrix response by the coincidence blurring factors, the IR system matrix containing PSF is finally established. Using this system matrix, the image is reconstructed by ordered subset expectation maximization (OSEM) algorithm. The experimental results show that the proposed system matrix can obviously improve the image radial resolution, contrast and noise property. Furthermore, the simulated single gamma-ray incidence response function only depends on the crystal configuration, so the method could be extended to any PET scanners with the same detector crystal configuration.

physics.med-ph↗