arXiv subjects

Liqun Fu

Publications and source records attributed to Liqun Fu.

At least 19 recordsLinked to original sources

arXiv2026

Achieving Rate-Concurrency Balance for Underwater Concurrent Random Access

Underwater acoustic networks face a fundamental rate--concurrency tradeoff: high-rate waveforms (e.g., OFDM, OTFS) are designed for point-to-point links and rely on orthogonal MAC protocols (e.g., TDMA) to avoid collisions, sacrificing concurrency; conversely, collision-resilient waveforms (e.g., CDMA, ZCMod) support uncoordinated access but are inherently rate-limited by spreading or sparse index modulation. We present \system, a cross-layer concurrent random-access system that combines two new components: (i) \textbf{EZCDM}, an equidistant ZC division-multiplexing waveform that activates multiple cyclic shifts of a ZC root as parallel sub-channels with a tunable rate--robustness tradeoff, and an intra-symbol differential receiver that eliminates the shared multipath channel response without explicit CIR estimation; and (ii) a \textbf{cross-layer link adaptation (LA) framework} featuring beacon-framed random access, user-specific closed-loop power control, and overlap- and CIR-aware common-MS selection. Channel-trace- and signal-trace-driven physical-layer experiments combined with PHY-in-the-loop network simulations demonstrate that \system\ achieves significant BER and throughput gains over conventional waveforms and MAC protocols by converting traditionally destructive collisions into decodable concurrent streams.

cs.NI↗

arXiv2026

Delay-Robust Deep Reinforcement Learning for Ranging-Free Channel Access under Mobility in Underwater Acoustic Networks

Long propagation delays in underwater acoustic networks (UWANs) cause spatio-temporal uncertainty, constraining channel utilization in medium access control (MAC) protocols. Node mobility within autonomous underwater vehicle scenarios exacerbates these challenges by introducing dynamic propagation delays and varying spatial topologies. We present MobiU-MAC, a deep reinforcement learning (DRL)-based MAC protocol for mobile node access in UWANs that maximizes throughput via autonomous learning. MobiU-MAC incorporates CHILL-STER, a novel DRL algorithm optimized for UWANs that is both ranging-free and delay-robust. CHILL-STER employs a credit horizon-limited $\lambda$-return (CHILL-Return) mechanism to achieve stable learning under asynchronous delayed rewards, while the companion spatio-temporal experience replay (STER) mechanism addresses topological changes arising from node mobility. This work also demonstrates theoretically that DRL attains optimal policy learning equivalent to a standard Markov decision process under long propagation delays without requiring ranging. Performance evaluations indicate that MobiU-MAC outperforms existing DRL-based MAC protocols for UWANs by leveraging the maximum system delay boundary without ranging overhead, supporting the effectiveness of the proposed theory and algorithm in complex underwater dynamic environments.

cs.NI↗

arXiv2026

Iterative Receiver Processing at Relays in PNC-Enabled Multi-Hop Underwater Acoustic Networks

Physical-layer network coding (PNC) can increase end-to-end throughput in bi-directional multi-hop underwater acoustic (UWA) networks. However, multipath delay spread and Doppler-induced inter-carrier interference (ICI) in UWA channels can degrade the reliability of PNC transmission in a three-node relay configuration. More critically, error accumulation across multiple relay nodes leads to a pronounced increase in the end-to-end bit error rate (BER) in multi-hop networks. To address this issue, we develop an iterative detection and decoding processing strategy for relay nodes within a PNC-enabled multi-hop UWA network based on orthogonal frequency division multiplexing (OFDM) modulation. The proposed design integrates three key algorithms: (i) an adaptive channel-aware factor graph detection algorithm that is suited for time-varying UWA channels; (ii) a parity-check-constrained soft-information refinement algorithm that improves the accuracy of the information feedback from the decoder to the detector; and (iii) a linear minimum mean square error (LMMSE) detection algorithm based on a superimposed model, which offers low computational complexity as an alternative scheme. Extensive simulation results demonstrate that the adaptive detection algorithm achieves BERs on the order of $10^{-5}$ at a relative velocity of 1.5 m/s UWA channel and a signal-to-noise (SNR) of 8~dB. Both lake experiments and sea trials in the Taiwan Strait confirm that the proposed iterative receiver algorithms outperform baseline schemes in terms of BER performance under practical UWA channel conditions, showing their robustness and applicability in real multi-hop deployments.

cs.NI↗

arXiv2026

Cross-Layer Traffic Allocation and Contention Window Optimization for Wi-Fi 7 MLO: When DRL Meets LSTM

To support future diverse applications, multi-link operation (MLO) has been introduced in the Wi-Fi 7 standard (IEEE 802.11be) to enable concurrent communication over multiple frequency bands. This new capability relies on a two-tier medium access control (MAC) architecture, where the upper MAC (U-MAC) allocates traffic across links and the lower MAC (L-MAC) performs independent channel access. However, MLO optimization is challenging due to the inherent coupling between the U-MAC and L-MAC, as well as the dynamic and complex nature of wireless networks. To address these challenges, we propose a cross-layer framework that jointly optimizes traffic allocation at the U-MAC layer and initial contention window (ICW) sizes at the L-MAC layer to maximize network throughput. Specifically, we extend the single-link Bianchi Markov model to develop an analytical framework that captures the relationship among network throughput, traffic allocation, and ICW sizes. Based on this framework, we formulate a nonconvex, nonlinear cross-layer optimization problem. To solve it efficiently, we design a long short-term memory-based soft actor-critic (LSTM-SAC) algorithm that leverages LSTM to handle the partial observability and non-Markovian dynamics inherent in Wi-Fi networks. Finally, using a well-developed event-based Wi-Fi simulator, we demonstrate that the proposed LSTM-SAC substantially outperforms existing benchmark solutions across a wide range of network settings.

cs.NI↗

arXiv2026

From Simulation to Reality: Practical Deep Reinforcement Learning-based Link Adaptation for Cellular Networks

Link Adaptation (LA) that dynamically adjusts the Modulation and Coding Schemes (MCS) to accommodate time-varying channels is crucial and challenging in cellular networks. Deep reinforcement learning (DRL)-based LA that learns to make decision through the interaction with the environment is a promising approach to improve throughput. However, existing DRL-based LA algorithms are typically evaluated in simplified simulation environments, neglecting practical issues such as ACK/NACK feedback delay, retransmission and parallel hybrid automatic repeat request (HARQ). Moreover, these algorithms overlook the impact of DRL execution latency, which can significantly degrade system performance. To address these challenges, we propose Decoupling-DQN (DC-DQN), a new DRL framework that separates traditional DRL's coupled training and inference processes into two modules based on Deep Q Networks (DQN): a real-time inference module and an out-of-decision-loop training module. Based on this framework, we introduce a novel DRL-based LA algorithm, DC-DQN-LA. The algorithm incorporates practical considerations by designing state, action, and reward functions that account for feedback delays, parallel HARQ, and retransmissions. We implemented a prototype using USRP software-defined radios and srsRAN software. Experimental results demonstrate that DC-DQN-LA improves throughput by 40\% to 70\% in mobile scenario compared with baseline LA algorithms, while maintaining comparable block error rates, and can quickly adapt to environment changes in mobile-to-static scenario. These results highlight the efficiency and practicality of the proposed DRL-based LA algorithm.

cs.NI↗

arXiv2026

ELSA: Efficient LLM-Centric Split Aggregation for Privacy-Aware Hierarchical Federated Learning over the Network Edge

Training large language models (LLMs) at the network edge faces fundamental challenges arising from device resource constraints, severe data heterogeneity, and heightened privacy risks. To address these challenges, we propose ELSA (Efficient LLM-centric Split Aggregation), a novel framework that systematically integrates split learning (SL) and hierarchical federated learning (HFL) for distributed LLM fine-tuning over resource-constrained edge networks. ELSA introduces three key innovations. First, it employs a task-agnostic, behavior-aware client clustering mechanism that constructs semantic fingerprints using public probe inputs and symmetric Kullback-Leibler (KL) divergence, augmented by prediction-consistency trust scoring and latency-aware edge assignment to jointly mitigate data heterogeneity, device unreliability, and communication constraints. Second, it employs a resource-aware dynamic model splitting strategy to adaptively partition the LLM into three segments across clients and edge servers, with the cloud used only for adapter aggregation, enabling an effective balance between on-device computation cost and global convergence stability. Third, it incorporates a lightweight communication scheme based on computational sketches combined with semantic subspace orthogonal perturbation (SS-OP) to reduce communication overhead while mitigating privacy leakage during model exchanges across the network. Extensive experiments across diverse NLP tasks demonstrate that ELSA consistently outperforms state-of-the-art baselines in terms of adaptability, convergence behavior, and robustness, establishing a scalable and privacy-aware solution for edge-side LLM fine-tuning under resource constraints.

cs.LG↗

arXiv2025

FUSION: Forecast-Embedded Agent Scheduling with Service Incentive Optimization over Distributed Air-Ground Edge Networks

In this paper, we introduce a first-of-its-kind forecasting-driven, incentive-aware service provisioning framework for distributed air-ground integrated networks that explicitly accounts for human-machine coexistence. In our framework, vehicular-UAV agent pairs (APs) are proactively dispatched to overloaded hotspots to augment the computing capacity of edge servers (ESs), which in turn gives rise to a set of challenges that we jointly address: highly uncertain spatio-temporal workloads, spatio-temporal coupling between road traffic and UAV capacity, forecast-driven contracting risks, and heterogeneous quality-of-service (QoS) requirements of human users (HUs) and machine users (MUs). To address these challenges, we propose FUSION, a two-stage optimization framework, consisting of an offline stage and an online stage. In the offline stage, a liquid neural network-powered module performs multi-step spatio-temporal demand forecasting at distributed ESs, whose outputs are exploited by an enhanced ant colony optimization-based routing scheme and an auction-based incentive-aware contracting mechanism, to jointly determine ES-AP contracts and pre-planned service routes. In the online stage, we model congestion-aware task scheduling as a service demander-side exact-potential game, where HUs and MUs select either local execution or accessible ES/UAV resources, and develop a potential-guided best-response dynamics algorithm that converges to an \varepsilon-NE under a positive improvement threshold and to a pure-strategy Nash equilibrium when the threshold is zero. Experiments on both synthetic and real-world datasets show that FUSION consistently achieves higher social welfare and improved resource utilization, while maintaining latency and energy costs comparable to state-of-the-art baselines and preserving individual rationality, budget balance, and near-truthfulness.

cs.NI↗

arXiv2025

PAST: Pilot and Adaptive Orchestration for Timely and Resilient Service Delivery in Edge-Assisted UAV Networks under Spatio-Temporal Dynamics

Incentive-driven resource trading is essential for uncrewed aerial vehicle (UAV) applications with intensive, time-sensitive computing demands. Traditional spot trading suffers from negotiation delays and high energy costs, while conventional futures trading struggles to adapt to the dynamic, uncertain UAV-edge environment. To address these challenges, we propose PAST (pilot-and-adaptive stable trading), a novel framework for edge-assisted UAV networks with spatio-temporal dynamism. PAST integrates two complementary mechanisms: PilotAO (pilot trading agreements with overbooking), a risk-aware, overbooking-enabled early-stage decision-making module that establishes long-term, mutually beneficial agreements and boosts resource utilization; and AdaptAO (adaptive trading agreements with overbooking rate update), an intelligent adaptation module that dynamically updates agreements and overbooking rates based on UAV mobility, supply-demand variations, and agreement performance. Together, these mechanisms integrate stable advance agreement construction with execution-feedback-driven adaptive renewal. Under the stated modeling assumptions, PilotAO satisfies the adopted individual-rationality and strong-stability criteria, as well as the competitive equilibrium and weak Pareto properties established for the considered trading model. Comprehensive experiments further demonstrate that, by coordinating advance planning with selective agreement renewal, PAST effectively balances stability, adaptability, and decision-making efficiency under dynamic UAV-edge conditions.

cs.DC↗

arXiv2025

Oh-Trust: Overbooking and Hybrid Trading-empowered Resource Scheduling with Smart Reputation Update over Dynamic Edge Networks

Incentive-driven computing resource sharing is crucial for meeting the ever-growing demands of emerging mobile applications. Although conventional spot trading offers a solution, it frequently leads to excessive overhead due to the need for real-time trading related interactions. Likewise, traditional futures trading, which depends on historical data, is susceptible to risks from network dynamics. This paper explores a dynamic and uncertain edge network comprising a computing platform, e.g., an edge server, that offers computing services as resource seller, and various types of mobile users with diverse resource demands as buyers, including fixed buyers (FBs) and uncertain occasional buyers (OBs) with fluctuating needs. To facilitate efficient and timely computing services, we propose an overbooking- and hybrid trading-empowered resource scheduling mechanism with reputation update, termed Oh-Trust. Particularly, our Oh-Trust incentivizes FBs to enter futures trading by signing long-term contracts with the seller, while simultaneously attracting OBs to spot trading, enhancing resource utilization and profitability for both parties. Crucially, to adapt to market fluctuations, a smart reputation updating mechanism is integrated, allowing for the timely renewal of long-term contracts to optimize trading performance. Extensive simulations using real-world datasets demonstrate the effectiveness of Oh-Trust across multiple evaluation metrics.

cs.NI↗

arXiv2025

Neural Channel Knowledge Map Assisted Scheduling Optimization of Active IRSs in Multi-User Systems

Intelligent Reflecting Surfaces (IRSs) have potential for significant performance gains in next-generation wireless networks but face key challenges, notably severe double-pathloss and complex multi-user scheduling due to hardware constraints. Active IRSs partially address pathloss but still require efficient scheduling in cell-level multi-IRS multi-user systems, whereby the overhead/delay of channel state acquisition and the scheduling complexity both rise dramatically as the user density and channel dimensions increase. Motivated by these challenges, this paper proposes a novel scheduling framework based on neural Channel Knowledge Map (CKM), designing Transformer-based deep neural networks (DNNs) to predict ergodic spectral efficiency (SE) from historical channel/throughput measurements tagged with user positions. Specifically, two cascaded networks, LPS-Net and SE-Net, are designed to predict link power statistics (LPS) and ergodic SE accurately. We further propose a low-complexity Stable Matching-Iterative Balancing (SM-IB) scheduling algorithm. Numerical evaluations verify that the proposed neural CKM significantly enhances prediction accuracy and computational efficiency, while the SM-IB algorithm effectively achieves near-optimal max-min throughput with greatly reduced complexity.

cs.IT↗

arXiv2025

Intelligent Channel Allocation for IEEE 802.11be Multi-Link Operation: When MAB Meets LLM

WiFi networks have achieved remarkable success in enabling seamless communication and data exchange worldwide. The IEEE 802.11be standard, known as WiFi 7, introduces Multi-Link Operation (MLO), a groundbreaking feature that enables devices to establish multiple simultaneous connections across different bands and channels. While MLO promises substantial improvements in network throughput and latency reduction, it presents significant challenges in channel allocation, particularly in dense network environments. Current research has predominantly focused on performance analysis and throughput optimization within static WiFi 7 network configurations. In contrast, this paper addresses the dynamic channel allocation problem in dense WiFi 7 networks with MLO capabilities. We formulate this challenge as a combinatorial optimization problem, leveraging a novel network performance analysis mechanism. Given the inherent lack of prior network information, we model the problem within a Multi-Armed Bandit (MAB) framework to enable online learning of optimal channel allocations. Our proposed Best-Arm Identification-enabled Monte Carlo Tree Search (BAI-MCTS) algorithm includes rigorous theoretical analysis, providing upper bounds for both sample complexity and error probability. To further reduce sample complexity and enhance generalizability across diverse network scenarios, we put forth LLM-BAI-MCTS, an intelligent algorithm for the dynamic channel allocation problem by integrating the Large Language Model (LLM) into the BAI-MCTS algorithm. Numerical results demonstrate that the BAI-MCTS algorithm achieves a convergence rate approximately $50.44\%$ faster than the state-of-the-art algorithms when reaching $98\%$ of the optimal value. Notably, the convergence rate of the LLM-BAI-MCTS algorithm increases by over $63.32\%$ in dense networks.

cs.NI↗

arXiv2025

Quasi-Static IRS: 3D Shaped Beamforming for Area Coverage Enhancement

Intelligent reflecting surface (IRS) is a promising paradigm to reconfigure the wireless environment for enhanced communication coverage and quality. However, to compensate for the double pathloss effect, massive IRS elements are required, raising concerns on the scalability of cost and complexity. This paper introduces a new architecture of quasi-static IRS (QS-IRS), which tunes element phases via mechanical adjustment or manually re-arranging the array topology. QS-IRS relies on massive production/assembly of purely passive elements only, and thus is suitable for ultra low-cost and large-scale deployment to enhance long-term coverage. To achieve this end, an IRS-aided area coverage problem is formulated, which explicitly considers the element radiation pattern (ERP), with the newly introduced shape masks for the mainlobe, and the sidelobe constraints to reduce energy leakage. An alternating optimization (AO) algorithm based on the difference-of-convex (DC) and successive convex approximation (SCA) procedure is proposed, which achieves shaped beamforming with power gains close to that of the joint optimization algorithm, but with significantly reduced computational complexity.

cs.IT↗

arXiv2025

Adaptive UAV-Assisted Hierarchical Federated Learning: Optimizing Energy, Latency, and Resilience for Dynamic Smart IoT

Hierarchical Federated Learning (HFL) extends conventional Federated Learning (FL) by introducing intermediate aggregation layers, enabling distributed learning in geographically dispersed environments, particularly relevant for smart IoT systems, such as remote monitoring and battlefield operations, where cellular connectivity is limited. In these scenarios, UAVs serve as mobile aggregators, dynamically connecting terrestrial IoT devices. This paper investigates an HFL architecture with energy-constrained, dynamically deployed UAVs prone to communication disruptions. We propose a novel approach to minimize global training costs by formulating a joint optimization problem that integrates learning configuration, bandwidth allocation, and device-to-UAV association, ensuring timely global aggregation before UAV disconnections and redeployments. The problem accounts for dynamic IoT devices and intermittent UAV connectivity and is NP-hard. To tackle this, we decompose it into three subproblems: \textit{(i)} optimizing learning configuration and bandwidth allocation via an augmented Lagrangian to reduce training costs; \textit{(ii)} introducing a device fitness score based on data heterogeneity (via Kullback-Leibler divergence), device-to-UAV proximity, and computational resources, using a TD3-based algorithm for adaptive device-to-UAV assignment; \textit{(iii)} developing a low-complexity two-stage greedy strategy for UAV redeployment and global aggregator selection, ensuring efficient aggregation despite UAV disconnections. Experiments on diverse real-world datasets validate the approach, demonstrating cost reduction and robust performance under communication disruptions.

cs.LG↗

arXiv2025

Future Resource Bank for ISAC: Achieving Fast and Stable Win-Win Matching for Both Individuals and Coalitions

Future wireless networks must support emerging applications where environmental awareness is as critical as data transmission. Integrated Sensing and Communication (ISAC) enables this vision by allowing base stations (BSs) to allocate bandwidth and power to mobile users (MUs) for communications and cooperative sensing. However, this resource allocation is highly challenging due to: (i) dynamic resource demands from MUs and resource supply from BSs, and (ii) the selfishness of MUs and BSs. To address these challenges, existing solutions rely on either real-time (online) resource trading, which incurs high overhead and failures, or static long-term (offline) resource contracts, which lack flexibility. To overcome these limitations, we propose the Future Resource Bank for ISAC, a hybrid trading framework that integrates offline and online resource allocation through a level-wise client model, where MUs and their coalitions negotiate with BSs. We introduce two mechanisms: (i) Role-Friendly Win-Win Matching (offRFW$^2$M), leveraging overbooking to establish risk-aware, stable contracts, and (ii) Effective Backup Win-Win Matching (onEBW$^2$M), which dynamically reallocates unmet demand and surplus supply. We theoretically prove stability, individual rationality, and weak Pareto optimality of these mechanisms. Through simulations, we show that our framework improves social welfare, latency, and energy efficiency compared to existing methods.

cs.DC↗

arXiv2025

Accelerating Stable Matching between Workers and Spatial-Temporal Tasks for Dynamic MCS: A Stagewise Service Trading Approach

Designing effective incentive mechanisms in mobile crowdsensing (MCS) networks is crucial for engaging distributed mobile users (workers) to contribute heterogeneous data for various applications (tasks). In this paper, we propose a novel stagewise trading framework to achieve efficient and stable task-worker matching, explicitly accounting for task diversity (e.g., spatio-temporal limitations) and network dynamics inherent in MCS environments. This framework integrates both futures and spot trading stages. In the former, we introduce the \textbf{f}utures \textbf{t}rading-driven \textbf{s}table \textbf{m}atching and \textbf{p}re-\textbf{p}ath-\textbf{p}lanning mechanism (FT-SMP$^3$), which enables long-term task-worker assignment and pre-planning of workers' trajectories based on historical statistics and risk-aware analysis. In the latter, we develop the \textbf{s}pot \textbf{t}rading-driven \textbf{D}QN-based \textbf{p}ath \textbf{p}lanning and onsite \textbf{w}orker \textbf{r}ecruitment mechanism (ST-DP$^2$WR), which dynamically improves the practical utilities of tasks and workers by supporting real-time recruitment and path adjustment. We rigorously prove that the proposed mechanisms satisfy key economic and algorithmic properties, including stability, individual rationality, competitive equilibrium, and weak Pareto optimality. Extensive experiements further validate the effectiveness of our framework in realistic network settings, demonstrating superior performance in terms of service quality, computational efficiency, and decision-making overhead.

cs.DC↗

arXiv2024

Digital Twin Enhanced Deep Reinforcement Learning for Intelligent Omni-Surface Configurations in MU-MIMO Systems

Intelligent omni-surface (IOS) is a promising technique to enhance the capacity of wireless networks, by reflecting and refracting the incident signal simultaneously. Traditional IOS configuration schemes, relying on all sub-channels' channel state information and user equipments' mobility, are difficult to implement in complex realistic systems. Existing works attempt to address this issue employing deep reinforcement learning (DRL), but this method requires a lot of trial-and-error interactions with the external environment for efficient results and thus cannot satisfy the real-time decision-making. To enable model-free and real-time IOS control, this paper puts forth a new framework that integrates DRL and digital twins. DeepIOS, a DRL based IOS configuration scheme with the goal of maximizing the sum data rate, is first developed to jointly optimize the phase-shift and amplitude of IOS in multi-user multiple-input-multiple-output systems. Thereafter, to further reduce the computational complexity, DeepIOS introduces an action branch architecture, which separately decides two optimization variables in parallel. Finally, a digital twin module is constructed through supervised learning as a pre-verification platform for DeepIOS, such that the decision-making's real-time can be guaranteed. The formulated framework is a closed-loop system, in which the physical space provides data to establish and calibrate the digital space, while the digital space generates experience samples for DeepIOS training and sends the trained parameters to the IOS controller for configurations. Numerical results show that compared with random and MAB schemes, the proposed framework attains a higher data rate and is more robust to different settings. Furthermore, the action branch architecture reduces DeepIOS's computational complexity, and the digital twin module improves the convergence speed and run-time.

cs.NI↗

arXiv2024

Integrated Sensing and Communications for Low-Altitude Economy: A Deep Reinforcement Learning Approach

This paper studies an integrated sensing and communications (ISAC) system for low-altitude economy (LAE), where a ground base station (GBS) provides communication and navigation services for authorized unmanned aerial vehicles (UAVs), while sensing the low-altitude airspace to monitor the unauthorized mobile target. The expected communication sum-rate over a given flight period is maximized by jointly optimizing the beamforming at the GBS and UAVs' trajectories, subject to the constraints on the average signal-to-noise ratio requirement for sensing, the flight mission and collision avoidance of UAVs, as well as the maximum transmit power at the GBS. Typically, this is a sequential decision-making problem with the given flight mission. Thus, we transform it to a specific Markov decision process (MDP) model called episode task. Based on this modeling, we propose a novel LAE-oriented ISAC scheme, referred to as Deep LAE-ISAC (DeepLSC), by leveraging the deep reinforcement learning (DRL) technique. In DeepLSC, a reward function and a new action selection policy termed constrained noise-exploration policy are judiciously designed to fulfill various constraints. To enable efficient learning in episode tasks, we develop a hierarchical experience replay mechanism, where the gist is to employ all experiences generated within each episode to jointly train the neural network. Besides, to enhance the convergence speed of DeepLSC, a symmetric experience augmentation mechanism, which simultaneously permutes the indexes of all variables to enrich available experience sets, is proposed. Simulation results demonstrate that compared with benchmarks, DeepLSC yields a higher sum-rate while meeting the preset constraints, achieves faster convergence, and is more robust against different settings.

cs.NI↗

arXiv2024

A Federated Online Restless Bandit Framework for Cooperative Resource Allocation

Restless multi-armed bandits (RMABs) have been widely utilized to address resource allocation problems with Markov reward processes (MRPs). Existing works often assume that the dynamics of MRPs are known prior, which makes the RMAB problem solvable from an optimization perspective. Nevertheless, an efficient learning-based solution for RMABs with unknown system dynamics remains an open problem. In this paper, we study the cooperative resource allocation problem with unknown system dynamics of MRPs. This problem can be modeled as a multi-agent online RMAB problem, where multiple agents collaboratively learn the system dynamics while maximizing their accumulated rewards. We devise a federated online RMAB framework to mitigate the communication overhead and data privacy issue by adopting the federated learning paradigm. Based on this framework, we put forth a Federated Thompson Sampling-enabled Whittle Index (FedTSWI) algorithm to solve this multi-agent online RMAB problem. The FedTSWI algorithm enjoys a high communication and computation efficiency, and a privacy guarantee. Moreover, we derive a regret upper bound for the FedTSWI algorithm. Finally, we demonstrate the effectiveness of the proposed algorithm on the case of online multi-user multi-channel access. Numerical results show that the proposed algorithm achieves a fast convergence rate of $\mathcal{O}(\sqrt{T\log(T)})$ and better performance compared with baselines. More importantly, its sample complexity decreases with the number of agents.

cs.LG↗