arXiv subjects

Han Yu

Publications and source records attributed to Han Yu.

At least 181 records · Page 10Linked to original sources

arXiv2022

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

Recent advances in NLP are brought by a range of large-scale pretrained language models (PLMs). These PLMs have brought significant performance gains for a range of NLP tasks, circumventing the need to customize complex designs for specific tasks. However, most current work focus on finetuning PLMs on a domain-specific datasets, ignoring the fact that the domain gap can lead to overfitting and even performance drop. Therefore, it is practically important to find an appropriate method to effectively adapt PLMs to a target domain of interest. Recently, a range of methods have been proposed to achieve this purpose. Early surveys on domain adaptation are not suitable for PLMs due to the sophisticated behavior exhibited by PLMs from traditional models trained from scratch and that domain adaptation of PLMs need to be redesigned to take effect. This paper aims to provide a survey on these newly proposed methods and shed light in how to apply traditional machine learning methods to newly evolved and future technologies. By examining the issues of deploying PLMs for downstream tasks, we propose a taxonomy of domain adaptation approaches from a machine learning system view, covering methods for input augmentation, model optimization and personalization. We discuss and compare those methods and suggest promising future research directions.

cs.CL↗

arXiv2022

Multi-Resource Allocation for On-Device Distributed Federated Learning Systems

This work poses a distributed multi-resource allocation scheme for minimizing the weighted sum of latency and energy consumption in the on-device distributed federated learning (FL) system. Each mobile device in the system engages the model training process within the specified area and allocates its computation and communication resources for deriving and uploading parameters, respectively, to minimize the objective of system subject to the computation/communication budget and a target latency requirement. In particular, mobile devices are connect via wireless TCP/IP architectures. Exploiting the optimization problem structure, the problem can be decomposed to two convex sub-problems. Drawing on the Lagrangian dual and harmony search techniques, we characterize the global optimal solution by the closed-form solutions to all sub-problems, which give qualitative insights to multi-resource tradeoff. Numerical simulations are used to validate the analysis and assess the performance of the proposed algorithm.

eess.SY↗

arXiv2022

Improving the Sample Efficiency of Prompt Tuning with Domain Adaptation

Prompt tuning, or the conditioning of a frozen pretrained language model (PLM) with soft prompts learned from data, has demonstrated impressive performance on a wide range of NLP tasks. However, prompt tuning requires a large training dataset to be effective and is outperformed by finetuning the entire PLM in data-scarce regimes. Previous work (Gu et al., 2022, Vu et al., 2022) proposed to transfer soft prompts pretrained on the source domain to the target domain. In this paper, we explore domain adaptation for prompt tuning, a problem setting where unlabeled data from the target domain are available during pretraining. We propose bOosting Prompt TunIng with doMain Adaptation (OPTIMA), which regularizes the decision boundary to be smooth around regions where source and target data distributions are similar. Extensive experiments demonstrate that OPTIMA significantly enhances the transferability and sample-efficiency of prompt tuning compared to strong baselines. Moreover, in few-shot settings, OPTIMA exceeds full-model tuning by a large margin.

cs.CL↗

arXiv2022

Measure-Theoretic Probability of Complex Co-occurrence and E-Integral

Complex high-dimensional co-occurrence data are increasingly popular from a complex system of interacting physical, biological and social processes in discretely indexed modifiable areal units or continuously indexed locations of a study region for landscape-based mechanism. Modeling, predicting and interpreting complex co-occurrences are very general and fundamental problems of statistical and machine learning in a broad variety of real-world modern applications. Probability and conditional probability of co-occurrence are introduced by being defined in a general setting with set functions to develop a rigorous measure-theoretic foundation for the inherent challenge of data sparseness. The data sparseness is a main challenge inherent to probabilistic modeling and reasoning of co-occurrence in statistical inference. The behavior of a class of natural integrals called E-integrals is investigated based on the defined conditional probability of co-occurrence. The results on the properties of E-integral are presented. The paper offers a novel measure-theoretic framework where E-integral as a basic measure-theoretic concept can be the starting point for the expectation functional approach preferred by Whittle (1992) and Pollard (2001) to the development of probability theory for the inherent challenge of co-occurrences emerging in modern high-dimensional co-occurrence data problems and opens the doors to more sophisticated and interesting research in complex high-dimensional co-occurrence data science.

stat.ML↗

arXiv2022

Empirical Evaluation of Data Augmentations for Biobehavioral Time Series Data with Deep Learning

Deep learning has performed remarkably well on many tasks recently. However, the superior performance of deep models relies heavily on the availability of a large number of training data, which limits the wide adaptation of deep models on various clinical and affective computing tasks, as the labeled data are usually very limited. As an effective technique to increase the data variability and thus train deep models with better generalization, data augmentation (DA) is a critical step for the success of deep learning models on biobehavioral time series data. However, the effectiveness of various DAs for different datasets with different tasks and deep models is understudied for biobehavioral time series data. In this paper, we first systematically review eight basic DA methods for biobehavioral time series data, and evaluate the effects on seven datasets with three backbones. Next, we explore adapting more recent DA techniques (i.e., automatic augmentation, random augmentation) to biobehavioral time series data by designing a new policy architecture applicable to time series data. Last, we try to answer the question of why a DA is effective (or not) by first summarizing two desired attributes for augmentations (challenging and faithful), and then utilizing two metrics to quantitatively measure the corresponding attributes, which can guide us in the search for more effective DA for biobehavioral time series data by designing more challenging but still faithful transformations. Our code and results are available at Link.

cs.LG↗

arXiv2022

Cooperative Resource Management in Quantum Key Distribution (QKD) Networks for Semantic Communication

Increasing privacy and security concerns in intelligence-native 6G networks require quantum key distribution-secured semantic information communication (QKD-SIC). In QKD-SIC systems, edge devices connected via quantum channels can efficiently encrypt semantic information from the semantic source, and securely transmit the encrypted semantic information to the semantic destination. In this paper, we consider an efficient resource (i.e., QKD and KM wavelengths) sharing problem to support QKD-SIC systems under the uncertainty of semantic information generated by edge devices. In such a system, QKD service providers offer QKD services with different subscription options to the edge devices. As such, to reduce the cost for the edge device users, we propose a QKD resource management framework for the edge devices communicating semantic information. The framework is based on a two-stage stochastic optimization model to achieve optimal QKD deployment. Moreover, to reduce the deployment cost of QKD service providers, QKD resources in the proposed framework can be utilized based on efficient QKD-SIC resource management, including semantic information transmission among edge devices, secret-key provisioning, and cooperation formation among QKD service providers. In detail, the formulated two-stage stochastic optimization model can achieve the optimal QKD-SIC resource deployment while meeting the secret-key requirements for semantic information transmission of edge devices. Moreover, to share the cost of the QKD resource pool among cooperative QKD service providers forming a coalition in a fair and interpretable manner, the proposed framework leverages the concept of Shapley value from cooperative game theory as a solution. Experimental results demonstrate that the proposed framework can reduce the deployment cost by about 40% compared with existing non-cooperative baselines.

cs.CR↗

arXiv2022

Towards Trustworthy AI-Empowered Real-Time Bidding for Online Advertisement Auctioning

Artificial intelligence-empowred Real-Time Bidding (AIRTB) is regarded as one of the most enabling technologies for online advertising. It has attracted significant research attention from diverse fields such as pattern recognition, game theory and mechanism design. Despite of its remarkable development and deployment, the AIRTB system can sometimes harm the interest of its participants (e.g., depleting the advertisers' budget with various kinds of fraud). As such, building trustworthy AIRTB auctioning systems has emerged as an important direction of research in this field in recent years. Due to the highly interdisciplinary nature of this field and a lack of a comprehensive survey, it is a challenge for researchers to enter this field and contribute towards building trustworthy AIRTB technologies. This paper bridges this important gap in trustworthy AIRTB literature. We start by analysing the key concerns of various AIRTB stakeholders and identify three main dimensions of trust building in AIRTB, namely security, robustness and fairness. For each of these dimensions, we propose a unique taxonomy of the state of the art, trace the root causes of possible breakdown of trust, and discuss the necessity of the given dimension. This is followed by a comprehensive review of existing strategies for fulfilling the requirements of each trust dimension. In addition, we discuss the promising future directions of research essential towards building trustworthy AIRTB systems to benefit the field of online advertising.

cs.IR↗

arXiv2022

Optimal Stochastic Resource Allocation for Distributed Quantum Computing

With the advent of interconnected quantum computers, i.e., distributed quantum computing (DQC), multiple quantum computers can now collaborate via quantum networks to perform massively complex computational tasks. However, DQC faces problems sharing quantum information because it cannot be cloned or duplicated between quantum computers. Thanks to advanced quantum mechanics, quantum computers can teleport quantum information across quantum networks. However, challenges to utilizing efficiently quantum resources, e.g., quantum computers and quantum channels, arise in DQC due to their capabilities and properties, such as uncertain qubit fidelity and quantum channel noise. In this paper, we propose a resource allocation scheme for DQC based on stochastic programming to minimize the total deployment cost for quantum resources. Essentially, the two-stage stochastic programming model is formulated to handle the uncertainty of quantum computing demands, computing power, and fidelity in quantum networks. The performance evaluation demonstrates the effectiveness and ability of the proposed scheme to balance the utilization of quantum computers and on-demand quantum computers while minimizing the overall cost of provisioning under uncertainty.

cs.DC↗

arXiv2022

DQC$^2$O: Distributed Quantum Computing for Collaborative Optimization in Future Networks

With the advantages of high-speed parallel processing, quantum computers can efficiently solve large-scale complex optimization problems in future networks. However, due to the uncertain qubit fidelity and quantum channel noise, distributed quantum computing which relies on quantum networks connected through entanglement faces a lot of challenges for exchanging information across quantum computers. In this paper, we propose an adaptive distributed quantum computing approach to manage quantum computers and quantum channels for solving optimization tasks in future networks. Firstly, we describe the fundamentals of quantum computing and its distributed concept in quantum networks. Secondly, to address the uncertainty of future demands of collaborative optimization tasks and instability over quantum networks, we propose a quantum resource allocation scheme based on stochastic programming for minimizing quantum resource consumption. Finally, based on the proposed approach, we discuss the potential applications for collaborative optimization in future networks, such as smart grid management, IoT cooperation, and UAV trajectory planning. Promising research directions that can lead to the design and implementation of future distributed quantum computing frameworks are also highlighted.

cs.DC↗

arXiv2022

Adaptive Resource Allocation in Quantum Key Distribution (QKD) for Federated Learning

Increasing privacy and security concerns in intelligence-native 6G networks require quantum key distribution-secured federated learning (QKD-FL), in which data owners connected via quantum channels can train an FL global model collaboratively without exposing their local datasets. To facilitate QKD-FL, the architectural design and routing management framework are essential. However, effective implementation is still lacking. To this end, we propose a hierarchical architecture for QKD-FL systems in which QKD resources (i.e., wavelengths) and routing are jointly optimized for FL applications. In particular, we focus on adaptive QKD resource allocation and routing for FL workers to minimize the deployment cost of QKD nodes under various uncertainties, including security requirements. The experimental results show that the proposed architecture and the resource allocation and routing model can reduce the deployment cost by 7.72\% compared to the CO-QBN algorithm.

cs.CR↗

arXiv2022

Robust Optimization with Decision-Dependent Information Discovery

Robust optimization is a popular paradigm for modeling and solving two- and multi-stage decision-making problems affected by uncertainty. In many real-world applications, the time of information discovery is decision-dependent and the uncertain parameters only become observable after an often costly investment. Yet, most of the literature assumes that uncertain parameters can be observed for free and that the sequence in which they are revealed is independent of the decision-maker's actions. To fill this gap. we consider two- and multi-stage robust optimization problems in which part of the decision variables control the time of information discovery. Thus, information can be discovered (at least in part) by making strategic exploratory investments in previous stages. We propose a novel dynamic formulation of the problem and prove its correctness. We leverage our model to provide a solution method inspired from the K-adaptability approximation. We reformulate the problem as a finite mixed-integer (resp. bilinear) program if none (resp. some of the) decision variables are real-valued. This finite program is solvable with off-the-shelf solvers. We generalize our approach to the minimization of piecewise linear convex functions. We demonstrate the effectiveness of our method in terms of interpretability, optimality, and speed on synthetic instances of the Pandora box problem, the preference elicitation problem, the best box problem, and the R&D project portfolio optimization problem. Finally, we evaluate it on an instance of the active preference elicitation problem used to recommend kidney allocation policies to policy-makers at the United Network for Organ Sharing.

math.OC↗

arXiv2022

Resource Allocation in Quantum Key Distribution (QKD) for Space-Air-Ground Integrated Networks

Space-air-ground integrated networks (SAGIN) are one of the most promising advanced paradigms in the sixth generation (6G) communication. SAGIN can support high data rates, low latency, and seamless network coverage for interconnected applications and services. However, communications in SAGIN are facing tremendous security threats from the ever-increasing capacity of quantum computers. Fortunately, quantum key distribution (QKD) for establishing secure communications in SAGIN, i.e., QKD over SAGIN, can provide information-theoretic security. To minimize the QKD deployment cost in SAGIN with heterogeneous nodes, in this paper, we propose a resource allocation scheme for QKD over SAGIN using stochastic programming. The proposed scheme is formulated via two-stage stochastic programming (SP), while considering uncertainties such as security requirements and weather conditions. Under extensive experiments, the results clearly show that the proposed scheme can achieve the optimal deployment cost under various security requirements and unpredictable weather conditions.

cs.CR↗

arXiv2022

Bias Reducing Multitask Learning on Mental Health Prediction

There has been an increase in research in developing machine learning models for mental health detection or prediction in recent years due to increased mental health issues in society. Effective use of mental health prediction or detection models can help mental health practitioners re-define mental illnesses more objectively than currently done, and identify illnesses at an earlier stage when interventions may be more effective. However, there is still a lack of standard in evaluating bias in such machine learning models in the field, which leads to challenges in providing reliable predictions and in addressing disparities. This lack of standards persists due to factors such as technical difficulties, complexities of high dimensional clinical health data, etc., which are especially true for physiological signals. This along with prior evidence of relations between some physiological signals with certain demographic identities restates the importance of exploring bias in mental health prediction models that utilize physiological signals. In this work, we aim to perform a fairness analysis and implement a multi-task learning based bias mitigation method on anxiety prediction models using ECG data. Our method is based on the idea of epistemic uncertainty and its relationship with model weights and feature space representation. Our analysis showed that our anxiety prediction base model introduced some bias with regards to age, income, ethnicity, and whether a participant is born in the U.S. or not, and our bias mitigation method performed better at reducing the bias in the model, when compared to the reweighting mitigation technique. Our analysis on feature importance also helped identify relationships between heart rate variability and multiple demographic groupings.

cs.LG↗

arXiv2022

NICO++: Towards Better Benchmarking for Domain Generalization

Despite the remarkable performance that modern deep neural networks have achieved on independent and identically distributed (I.I.D.) data, they can crash under distribution shifts. Most current evaluation methods for domain generalization (DG) adopt the leave-one-out strategy as a compromise on the limited number of domains. We propose a large-scale benchmark with extensive labeled domains named NICO++ along with more rational evaluation methods for comprehensively evaluating DG algorithms. To evaluate DG datasets, we propose two metrics to quantify covariate shift and concept shift, respectively. Two novel generalization bounds from the perspective of data construction are proposed to prove that limited concept shift and significant covariate shift favor the evaluation capability for generalization. Through extensive experiments, NICO++ shows its superior evaluation capability compared with current DG datasets and its contribution in alleviating unfairness caused by the leak of oracle knowledge in model selection.

cs.CV↗

arXiv2022

A note on dyadic approximation in Cantor's set

We consider the convergence theory for dyadic approximation in the middle-third Cantor set, $K$, for approximation functions of the form $ψ_τ(n) = n^{-τ}$ ($τ\ge 0$). In particular, we show that for values of $τ$ beyond a certain threshold we have that almost no point in $K$ is dyadically $ψ_τ$-well approximable with respect to the natural probability measure on $K$. This refines a previous result in this direction obtained by the first, third, and fourth named authors (arXiv, 2020).

math.NT↗

arXiv2022

Federated Learning for Personalized Humor Recognition

Computational understanding of humor is an important topic under creative language understanding and modeling. It can play a key role in complex human-AI interactions. The challenge here is that human perception of humorous content is highly subjective. The same joke may receive different funniness ratings from different readers. This makes it highly challenging for humor recognition models to achieve personalization in practical scenarios. Existing approaches are generally designed based on the assumption that users have a consensus on whether a given text is humorous or not. Thus, they cannot handle diverse humor preferences well. In this paper, we propose the FedHumor approach for the recognition of humorous content in a personalized manner through Federated Learning (FL). Extending a pre-trained language model, FedHumor guides the fine-tuning process by considering diverse distributions of humor preferences from individuals. It incorporates a diversity adaptation strategy into the FL paradigm to train a personalized humor recognition model. To the best of our knowledge, FedHumor is the first text-based personalized humor recognition model through federated learning. Extensive experiments demonstrate the advantage of FedHumor in recognizing humorous texts compared to nine state-of-the-art humor recognition approaches with superior capability for handling the diversity in humor labels produced by users with diverse preferences.

cs.CL↗

arXiv2022

Towards Personalized Federated Learning

In parallel with the rapid adoption of Artificial Intelligence (AI) empowered by advances in AI research, there have been growing awareness and concerns of data privacy. Recent significant developments in the data regulation landscape have prompted a seismic shift in interest towards privacy-preserving AI. This has contributed to the popularity of Federated Learning (FL), the leading paradigm for the training of machine learning models on data silos in a privacy-preserving manner. In this survey, we explore the domain of Personalized FL (PFL) to address the fundamental challenges of FL on heterogeneous data, a universal characteristic inherent in all real-world datasets. We analyze the key motivations for PFL and present a unique taxonomy of PFL techniques categorized according to the key challenges and personalization strategies in PFL. We highlight their key ideas, challenges and opportunities and envision promising future trajectories of research towards new PFL architectural design, realistic PFL benchmarking, and trustworthy PFL approaches.

cs.LG↗

arXiv2022

MarS-FL: Enabling Competitors to Collaborate in Federated Learning

Federated learning (FL) is rapidly gaining popularity and enables multiple data owners ({\em a.k.a.} FL participants) to collaboratively train machine learning models in a privacy-preserving way. A key unaddressed scenario is that these FL participants are in a competitive market, where market shares represent their competitiveness. Although they are interested to enhance the performance of their respective models through FL, market leaders (who are often data owners who can contribute significantly to building high performance FL models) want to avoid losing their market shares by enhancing their competitors' models. Currently, there is no modeling tool to analyze such scenarios and support informed decision-making. In this paper, we bridge this gap by proposing the \underline{mar}ket \underline{s}hare-based decision support framework for participation in \underline{FL} (MarS-FL). We introduce {\em two notions of $δ$-stable market} and {\em friendliness} to measure the viability of FL and the market acceptability of FL. The FL participants' behaviours can then be predicted using game theoretic tools (i.e., their optimal strategies concerning participation in FL). If the market $δ$-stability is achievable, the final model performance improvement of each FL-PT shall be bounded, which relates to the market conditions of FL applications. We provide tight bounds and quantify the friendliness, $κ$, of given market conditions to FL. Experimental results show the viability of FL in a wide range of market conditions. Our results are useful for identifying the market conditions under which collaborative FL model training is viable among competitors, and the requirements that have to be imposed while applying FL under these conditions.

cs.LG↗