arXiv subjects

Shengchao Zhu

Publications and source records attributed to Shengchao Zhu.

6 recordsLinked to original sources

arXiv2026

Diffusion-Guided Cooperative Policy Learning for Target Tracking Based on Underwater Mobile Agent Networks

Multi-agent reinforcement learning (MARL) provides a promising solution for cooperative target tracking in networks of autonomous underwater vehicles (AUVs). However, existing methods still face three major challenges: 1) policy non-stationarity caused by concurrent updates among multiple agents; 2) inefficient policy learning caused by the heterogeneous quality of experiences accumulated during exploration; and 3) policy drift between stochastic exploration and deterministic execution under dynamic underwater disturbances. To address these challenges, this paper develops a four-layer hierarchical MARL architecture comprising global training scheduling, multi-agent coordination, local policy generation, and real-time action execution. Building on this architecture, we propose a Supervised Diffusion-Aided MARL (SDA-MARL) algorithm with three closely coupled mechanisms. First, a dual-decision policy integrates a diffusion-based generative branch with a Deep Deterministic Policy Gradient (DDPG) branch, while segregated experience pools reduce training interference between the two branches. Second, a supervised sample-selection mechanism identifies high-quality tracking transitions and uses their actions to guide reverse diffusion, enabling the generative policy to concentrate on effective regions of the action space. Third, a behavioral-cloning loss transfers diffusion-generated actions to the deterministic DDPG Actor, thereby aligning exploration with execution and suppressing policy drift. Experiments conducted in six-degree-of-freedom underwater environments across multiple AUV-target configurations show that SDA-MARL achieves faster convergence, higher tracking accuracy, more consistent inter-AUV velocities, and shorter tracking paths than the compared MARL methods.

cs.NI↗

arXiv2026

Multi-AUV Ad-hoc network-based Target Tracking: A Value Gradient Guidance Multi-Agent Diffusion Reinforcement Learning Approach

Multi-AUV ad-hoc network-based target tracking requires networked autonomous underwater vehicles (AUVs) to cooperatively track maneuvering targets under constrained acoustic communication, dynamic topology, and uncertain ocean disturbances. Although multi-agent reinforcement learning (MARL) enables decentralized coordination through centralized training, existing methods suffer from high-dimensional joint state-action modeling, noise-sensitive policy generation, leading to unstable training and degraded tracking. To address these issues, we propose VGG-MADiffRL, a value-gradient-guided multi-agent diffusion RL algorithm, and MDCA, a diffusion?based hierarchical control architecture. Leveraging underwater mission characteristics, we model sonar detection mechanisms and ocean current disturbances, formulating cooperative tracking for multi-AUV ad-hoc networks as an MDP. The proposed MDCA constitutes a three-tier closed-loop control framework: a global intelligent control layer, a local online training layer, and a physical action execution layer. This structure enables synergistic optimization across task allocation, local decision processes, and execution feedback. Within MDCA, the local online training layer is the policy learning framework; VGG-MADiffRL builds on diffusion policies and incorporates value gradients to guide action generation in the reverse denoising process, steering the generated actions towards higher expected returns. It employs twin value networks with joint optimization and soft target updates to mitigate overestimation and training oscillations, promoting more stable convergence. Experimental results show that VGG-MADiffRL consistently achieves faster convergence, higher tracking accuracy, and smoother training dynamics in cooperative tracking scenarios, validating its effectiveness and practical engineering value in dynamic underwater settings.

cs.LG↗

arXiv2026

Task-Semantic Graph-Driven Distributed Agent Networking for Underwater Target Tracking

Autonomous underwater vehicle (AUV) swarms are emerging as intelligent underwater networks, where each node must sense, communicate, process local data, and make decisions under severe acoustic constraints. Persistent underwater target tracking is a typical task with moving targets, changing communication topology, intermittent acoustic links, and limited observation for each AUV. Multi-agent reinforcement learning (MARL) is a natural candidate for distributed tracking, yet existing studies still lack a unified open-source platform for evaluating different MARL algorithms under six-degree-of-freedom AUV dynamics. In addition, policies trained with raw geometric states and low-level force actions often struggle to represent task phases, observation reliability, link quality, and local cooperation roles. This paper addresses these issues by developing an open-source MARL-AUV platform that integrates DI-engine with a six-degree-of-freedom underwater AUV target-tracking simulator. To the best of our knowledge, it is the first open platform that connects a public MARL training framework with physically modeled AUV swarm-based tasks, and provides a unified experimental protocol for fair training, testing, and comparison of representative RL and MARL algorithms. Based on this platform, we propose STG-MAPPO, a Semantic Task Graph-enhanced variant of Multi-Agent Proximal Policy Optimization. STG-MAPPO builds semantic policy inputs from tracking diagnostics, task phases, observation confidence, link availability, neighbor tracking quality, and local role advantage. A compact semantic task graph links communication-constrained network states to decentralized actor decisions, and a velocity-level action abstraction maps high-level cooperative decisions to executable six-degree-offreedom AUV control inputs.The code is available at https://github.com/dasjsaj/MARL-AUV.

cs.RO↗

arXiv2026

Multi-AUV Ad-hoc Networks-Based Multi-Target Tracking Based on Scene-Adaptive Embodied Intelligence

With the rapid advancement of underwater net-working and multi-agent coordination technologies, autonomous underwater vehicle (AUV) ad-hoc networks have emerged as a pivotal framework for executing complex maritime missions, such as multi-target tracking. However, traditional data-centricarchitectures struggle to maintain operational consistency under highly dynamic topological fluctuations and severely constrained acoustic communication bandwidth. This article proposes a scene-adaptive embodied intelligence (EI) architecture for multi-AUV ad-hoc networks, which re-envisions AUVs as embodied entities by integrating perception, decision-making, and physical execution into a unified cognitive loop. To materialize the functional interaction between these layers, we define a beacon-based communication and control model that treats the communication link as a dynamic constraint-aware channel, effectively bridging the gap between high-level policy inference and decentralized physical actuation. Specifically, the proposed architecture employs a three-layer functional framework and introduces a Scene-Adaptive MARL (SA-MARL) algorithm featuring a dual-path critic mechanism. By integrating a scene critic network and a general critic network through a weight-based dynamic fusion process, SA-MARL effectively decouples specialized tracking tasks from global safety constraints, facilitating autonomous policy evolution. Evaluation results demonstrate that the proposedscheme significantly accelerates policy convergence and achieves superior tracking accuracy compared to mainstream MARL approaches, maintaining robust performance even under intense environmental interference and fluid topological shifts.

cs.RO↗

arXiv2025

Smart Interrupted Routing Based on Multi-head Attention Mask Mechanism-Driven MARL in Software-defined UASNs

Routing-driven timely data collection in Underwater Acoustic Sensor Networks (UASNs) is crucial for marine environmental monitoring, disaster warning and underwater resource exploration, etc. However, harsh underwater conditions, including high delays, limited bandwidth, and dynamic topologies - make efficient routing decisions challenging in UASNs. In this paper, we propose a smart interrupted routing scheme for UASNs to address dynamic underwater challenges. We first model underwater noise influences from real underwater routing features, e.g., turbulence and storms. We then propose a Software-Defined Networking (SDN)-based Interrupted Software-defined UASNs Reinforcement Learning (ISURL) framework which ensures adaptive routing through dynamically failure handling (e.g., energy depletion of sensor nodes or link instability) and real-time interrupted recovery. Based on ISURL, we propose MA-MAPPO algorithm, integrating multi-head attention mask mechanism with MAPPO to filter out infeasible actions and streamline training. Furthermore, to support interrupted data routing in UASNs, we introduce MA-MAPPO_i, MA-MAPPO with interrupted policy, to enable smart interrupted routing decision in UASNs. The evaluations demonstrate that our proposed routing scheme achieves exact underwater data routing decision with faster convergence speed and lower routing delays than existing approaches.

cs.NI↗

arXiv2024

Multi-AUV Cooperative Underwater Multi-Target Tracking Based on Dynamic-Switching-enabled Multi-Agent Reinforcement Learning

In recent years, autonomous underwater vehicle (AUV) swarms are gradually becoming popular and have been widely promoted in ocean exploration or underwater tracking, etc. In this paper, we propose a multi-AUV cooperative underwater multi-target tracking algorithm especially when the real underwater factors are taken into account. We first give normally modelling approach for the underwater sonar-based detection and the ocean current interference on the target tracking process. Then, based on software-defined networking (SDN), we regard the AUV swarm as a underwater ad-hoc network and propose a hierarchical software-defined multi-AUV reinforcement learning (HSARL) architecture. Based on the proposed HSARL architecture, we propose the "Dynamic-Switching" mechanism, it includes "Dynamic-Switching Attention" and "Dynamic-Switching Resampling" mechanisms which accelerate the HSARL algorithm's convergence speed and effectively prevents it from getting stuck in a local optimum state. Additionally, we introduce the reward reshaping mechanism for further accelerating the convergence speed of the proposed HSARL algorithm in early phase. Finally, based on a proposed AUV classification method, we propose a cooperative tracking algorithm called Dynamic-Switching-Based MARL (DSBM)-driven tracking algorithm. Evaluation results demonstrate that our proposed DSBM tracking algorithm can perform precise underwater multi-target tracking, comparing with many of recent research products in terms of various important metrics.

cs.MA↗