arXiv subjects

Ziang Zhang

Publications and source records attributed to Ziang Zhang.

35 records · Page 2Linked to original sources

arXiv2024

RIMformer: An End-to-End Transformer for FMCW Radar Interference Mitigation

Frequency-modulated continuous-wave (FMCW) radar plays a pivotal role in the field of remote sensing. The increasing degree of FMCW radar deployment has increased the mutual interference, which weakens the detection capabilities of radars and threatens reliability and safety of systems. In this paper, a novel FMCW radar interference mitigation (RIM) method, termed as RIMformer, is proposed by using an end-to-end Transformer-based structure. In the RIMformer, a dual multi-head self-attention mechanism is proposed to capture the correlations among the distinct distance elements of intermediate frequency (IF) signals. Additionally, an improved convolutional block is integrated to harness the power of convolution for extracting local features. The architecture is designed to process time-domain IF signals in an end-to-end manner, thereby avoiding the need for additional manual data processing steps. The improved decoder structure ensures the parallelization of the network to increase its computational efficiency. Simulation and measurement experiments are carried out to validate the accuracy and effectiveness of the proposed method. The results show that the proposed RIMformer can effectively mitigate interference and restore the target signals.

eess.SP↗

arXiv2024

OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces

Recently, human-computer interaction with various modalities has shown promising applications, like GPT-4o and Gemini. Given the foundational role of multimodal joint representation in understanding and generation pipelines, high-quality omni joint representations would be a step toward co-processing more diverse multimodal information. In this work, we present OmniBind, large-scale multimodal joint representation models ranging in scale from 7 billion to 30 billion parameters, which support 3D, audio, image, and language inputs. Due to the scarcity of data pairs across all modalities, instead of training large models from scratch, we propose remapping and binding the spaces of various pre-trained specialist models together. This approach enables "scaling up" by indirectly increasing the model parameters and the amount of seen data. To effectively integrate various spaces, we dynamically assign weights to different spaces by learning routers with two objectives: cross-modal overall alignment and language representation decoupling. Notably, since binding and routing spaces both only require lightweight networks, OmniBind is extremely training-efficient. Learning the largest 30B model requires merely unpaired unimodal data and approximately 3 days on a single 8-4090 node. Extensive experiments demonstrate the versatility and superiority of OmniBind as an omni representation model, highlighting its great potential for diverse applications, such as any-query and composable multimodal understanding.

cs.CV↗

arXiv2024

FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion

Unified multi-model representation spaces are the foundation of multimodal understanding and generation. However, the billions of model parameters and catastrophic forgetting problems make it challenging to further enhance pre-trained unified spaces. In this work, we propose FreeBind, an idea that treats multimodal representation spaces as basic units, and freely augments pre-trained unified space by integrating knowledge from extra expert spaces via "space bonds". Specifically, we introduce two kinds of basic space bonds: 1) Space Displacement Bond and 2) Space Combination Bond. Based on these basic bonds, we design Complex Sequential & Parallel Bonds to effectively integrate multiple spaces simultaneously. Benefiting from the modularization concept, we further propose a coarse-to-fine customized inference strategy to flexibly adjust the enhanced unified space for different purposes. Experimentally, we bind ImageBind with extra image-text and audio-text expert spaces, resulting in three main variants: ImageBind++, InternVL_IB, and InternVL_IB++. These resulting spaces outperform ImageBind on 5 audio-image-text downstream tasks across 9 datasets. Moreover, via customized inference, it even surpasses the advanced audio-text and image-text expert spaces.

cs.CV↗

arXiv2024

ProMamba: Prompt-Mamba for polyp segmentation

Detecting polyps through colonoscopy is an important task in medical image segmentation, which provides significant assistance and reference value for clinical surgery. However, accurate segmentation of polyps is a challenging task due to two main reasons. Firstly, polyps exhibit various shapes and colors. Secondly, the boundaries between polyps and their normal surroundings are often unclear. Additionally, significant differences between different datasets lead to limited generalization capabilities of existing methods. To address these issues, we propose a segmentation model based on Prompt-Mamba, which incorporates the latest Vision-Mamba and prompt technologies. Compared to previous models trained on the same dataset, our model not only maintains high segmentation accuracy on the validation part of the same dataset but also demonstrates superior accuracy on unseen datasets, exhibiting excellent generalization capabilities. Notably, we are the first to apply the Vision-Mamba architecture to polyp segmentation and the first to utilize prompt technology in a polyp segmentation model. Our model efficiently accomplishes segmentation tasks, surpassing previous state-of-the-art methods by an average of 5% across six datasets. Furthermore, we have developed multiple versions of our model with scaled parameter counts, achieving better performance than previous models even with fewer parameters. Our code and trained weights will be released soon.

cs.CV↗

arXiv2024

Bayesian Optimization Sequential Surrogate (BOSS) Algorithm: Fast Bayesian Inference for a Broad Class of Bayesian Hierarchical Models

Approximate Bayesian inference based on Laplace approximation and quadrature methods have become increasingly popular for their efficiency at fitting latent Gaussian models (LGM), which encompass popular models such as Bayesian generalized linear models, survival models, and spatio-temporal models. However, many useful models fall under the LGM framework only if some conditioning parameters are fixed, as the design matrix would vary with these parameters otherwise. Such models are termed the conditional LGMs with examples in change-point detection, non-linear regression, etc. Existing methods for fitting conditional LGMs rely on grid search or Markov-chain Monte Carlo (MCMC); both require a large number of evaluations of the unnormalized posterior density of the conditioning parameters. As each evaluation of the density requires fitting a separate LGM, these methods become computationally prohibitive beyond simple scenarios. In this work, we introduce the Bayesian optimization sequential surrogate (BOSS) algorithm, which combines Bayesian optimization with approximate Bayesian inference methods to significantly reduce the computational resources required for fitting conditional LGMs. With orders of magnitude fewer evaluations compared to grid or MCMC methods, Bayesian optimization provides us with sequential design points that capture the majority of the posterior mass of the conditioning parameters, which subsequently yields an accurate surrogate posterior distribution that can be easily normalized. We illustrate the efficiency, accuracy, and practical utility of the proposed method through extensive simulation studies and real-world applications in epidemiology, environmental sciences, and astrophysics.

stat.ME↗

arXiv2024

A Segmentation Foundation Model for Diverse-type Tumors

Large pre-trained models with their numerous model parameters and extensive training datasets have shown excellent performance in various tasks. Many publicly available medical image datasets do not have a sufficient amount of data so there are few large-scale models in medical imaging. We propose a large-scale Tumor Segmentation Foundation Model (TSFM) with 1.6 billion parameters using Resblock-backbone and Transformer-bottleneck,which has good transfer ability for downstream tasks. To make TSFM exhibit good performance in tumor segmentation, we make full use of the strong spatial correlation between tumors and organs in the medical image, innovatively fuse 7 tumor datasets and 3 multi-organ datasets to build a 3D medical dataset pool, including 2779 cases with totally 300k medical images, whose size currently exceeds many other single publicly available datasets. TSFM is the pre-trained model for medical image segmentation, which also can be transferred to multiple downstream tasks for fine-tuning learning. The average performance of our pre-trained model is 2% higher than that of nnU-Net across various tumor types. In the transfer learning task, TSFM only needs 5% training epochs of nnU-Net to achieve similar performance and can surpass nnU-Net by 2% on average with 10% training epoch. Pre-trained TSFM and its code will be released soon.

eess.IV↗

arXiv2023

Connecting Multi-modal Contrastive Representations

Multi-modal Contrastive Representation learning aims to encode different modalities into a semantically aligned shared space. This paradigm shows remarkable generalization ability on numerous downstream tasks across various modalities. However, the reliance on massive high-quality data pairs limits its further development on more modalities. This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR). Specifically, given two existing MCRs pre-trained on (A, B) and (B, C) modality pairs, we project them to a new space and use the data from the overlapping modality B to aligning the two MCRs in the new space. Meanwhile, since the modality pairs (A, B) and (B, C) are already aligned within each MCR, the connection learned by overlapping modality can also be transferred to non-overlapping modality pair (A, C). To unleash the potential of C-MCR, we further introduce a semantic-enhanced inter- and intra-MCR connection method. We first enhance the semantic consistency and completion of embeddings across different modalities for more robust alignment. Then we utilize the inter-MCR alignment to establish the connection, and employ the intra-MCR alignment to better maintain the connection for inputs from non-overlapping modalities. To demonstrate the effectiveness of C-MCR, we connect CLIP and CLAP via texts to derive audio-visual representations, and integrate CLIP and ULIP via images for 3D-language representations. Remarkably, without using any paired data, C-MCR for audio-visual achieves state-of-the-art performance on audio-image retrieval, audio-visual source localization, and counterfactual audio-image recognition tasks. Furthermore, C-MCR for 3D-language also attains advanced zero-shot 3D point cloud classification accuracy on ModelNet40.

cs.LG↗

arXiv2023

Extending Multi-modal Contrastive Representations

Multi-modal contrastive representation (MCR) of more than three modalities is critical in multi-modal learning. Although recent methods showcase impressive achievements, the high dependence on large-scale, high-quality paired data and the expensive training costs limit their further development. Inspired by recent C-MCR, this paper proposes Extending Multimodal Contrastive Representation (Ex-MCR), a training-efficient and paired-data-free method to flexibly learn unified contrastive representation space for more than three modalities by integrating the knowledge of existing MCR spaces. Specifically, Ex-MCR aligns multiple existing MCRs into the same based MCR, which can effectively preserve the original semantic alignment of the based MCR. Besides, we comprehensively enhance the entire learning pipeline for aligning MCR spaces from the perspectives of training data, architecture, and learning objectives. With the preserved original modality alignment and the enhanced space alignment, Ex-MCR shows superior representation learning performance and excellent modality extensibility. To demonstrate the effectiveness of Ex-MCR, we align the MCR spaces of CLAP (audio-text) and ULIP (3D-vision) into the CLIP (vision-text), leveraging the overlapping text and image modality, respectively. Remarkably, without using any paired data, Ex-MCR learns a 3D-image-text-audio unified contrastive representation, and it achieves state-of-the-art performance on audio-visual, 3D-image, audio-text, visual-text retrieval, and 3D object classification tasks. More importantly, extensive qualitative results further demonstrate the emergent semantic alignment between the extended modalities (e.g., audio and 3D), which highlights the great potential of modality extensibility.

cs.CV↗

arXiv2023

Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes

3D scene understanding has gained significant attention due to its wide range of applications. However, existing methods for 3D scene understanding are limited to specific downstream tasks, which hinders their practicality in real-world applications. This paper presents Chat-3D, which combines the 3D visual perceptual ability of pre-trained 3D representations and the impressive reasoning and conversation capabilities of advanced LLMs to achieve the first universal dialogue systems for 3D scenes. Specifically, we align 3D representations into the feature space of LLMs, thus enabling LLMs to perceive the 3D world. Given the scarcity of 3D scene-text data, we propose a three-stage training strategy to efficiently utilize the available data for better alignment. To enhance the reasoning ability and develop a user-friendly interaction scheme, we further construct a high-quality object-centric 3D instruction dataset and design an associated object-centric prompt. Our experiments show that Chat-3D achieves an impressive ability to comprehend diverse instructions for 3D scenes, engage in intricate spatial reasoning, and incorporate external knowledge into its responses. Chat-3D achieves a 75.6% relative score compared with GPT-4 on the constructed instruction dataset.

cs.CV↗

arXiv2023

Efficient Modeling of Quasi-Periodic Data with Seasonal Gaussian Process

Quasi-periodicity refers to a pattern in a function where it appears periodic but has evolving amplitudes over time. This is often the case in practical settings such as the modeling of case counts of infectious disease or the carbon dioxide (CO2) concentration over time. In this paper, we introduce a class of Gaussian processes, called seasonal Gaussian Processes (sGP), for model-based inference of such quasi-periodic behavior. We illustrate that the exact sGP can be efficiently fit within $O(n)$ time using its state space representation for equally spaced locations. However, for large datasets with irregular spacing, the exact approach becomes computationally inefficient and unstable. To address this, we develop a continuous finite dimensional approximation for sGP using the seasonal B-spline (sB-spline) basis constructed by damping B-splines with sinusoidal functions. We prove that the proposed approximation converges in distribution to the true sGP as the number of basis functions increases, and show its superior approximation quality through numerical studies. We also provide a unified and interpretable way to define priors for the sGP, based on the notion of predictive standard deviation (PSD). Finally, we implement the proposed inference method on several real data examples to illustrate its practical usage.

stat.ME↗

arXiv2023

Discrete States-Based Trajectory Planning for Nonholonomic Robots

Due to nonholonomic dynamics, the motion planning of nonholonomic robots is always a difficult problem. This letter presents a Discrete States-based Trajectory Planning(DSTP) algorithm for autonomous nonholonomic robots. The proposed algorithm represents the trajectory as x and y positions, orientation angle, longitude velocity and acceleration, angular velocity, and time intervals. More variables make the expression of optimization and constraints simpler, reduce the error caused by too many approximations, and also handle the gear shifting situation. L-BFGS-B is used to deal with the optimization of many variables and box constraints, thus speeding up the problem solving. Various simulation experiments compared with prior works have validated that our algorithm has an order-of-magnitude efficiency advantage and can generate a smoother trajectory with a high speed and low control effort. Besides, real-world experiments are also conducted to verify the feasibility of our algorithm in real scenes. We will release our codes as ros packages.

cs.RO↗

arXiv2023

Model-based Smoothing with Integrated Wiener Processes and Overlapping Splines

In many applications that involve the inference of an unknown smooth function, the inference of its derivatives will often be just as important as that of the function itself. To make joint inferences of the function and its derivatives, a class of Gaussian processes called $p^{\text{th}}$ order Integrated Wiener's Process (IWP), is considered. Methods for constructing a finite element (FEM) approximation of an IWP exist but have focused only on the order $p = 2$ case which does not allow appropriate inference for derivatives, and their computational feasibility relies on additional approximation to the FEM itself. In this article, we propose an alternative FEM approximation, called overlapping splines (O-spline), which pursues computational feasibility directly through the choice of test functions, and mirrors the construction of an IWP as the Ospline results from the multiple integrations of these same test functions. The O-spline approximation applies for any order $p \in \mathbb{Z}^+$, is computationally efficient and provides consistent inference for all derivatives up to order $p-1$. It is shown both theoretically, and empirically through simulation, that the O-spline approximation converges to the true IWP as the number of knots increases. We further provide a unified and interpretable way to define priors for the smoothing parameter based on the notion of predictive standard deviation (PSD), which is invariant to the order $p$ and the placement of the knot. Finally, we demonstrate the practical use of the O-spline approximation through simulation studies and an analysis of COVID death rates where the inference is carried on both the function and its derivatives where the latter has an important interpretation in terms of the course of the pandemic.

stat.ME↗

arXiv2022

The hidden factor: accounting for covariate effects in power and sample size computation for a binary trait

Accurate power and sample size estimation are crucial to the design and analysis of genetic association studies. When analyzing a binary trait via logistic regression, important covariates such as age and sex are typically included in the model. However, their effects are rarely properly considered in power or sample size computation during study planning. Unlike when analyzing a continuous trait, the power of association testing between a binary trait and a genetic variant depends, explicitly, on covariate effects, even under the assumption of gene-environment independence. Earlier work recognizes this hidden factor but implemented methods are not flexible. We thus propose and implement a generalized method for estimating power and sample size for (discovery or replication) association studies of binary traits that a) accommodates different types of non-genetic covariates E, b) deals with different types of G-E relationships, and c) is computationally efficient. Extensive simulation studies show that the proposed method is accurate and computationally efficient for both prospective and retrospective sampling designs with various covariate structures. A proof-of-principle application focused on the understudied African sample in the UK Biobank data. Results show that, in contrast to studying the continuous blood pressure trait, when analyzing the binary hypertension trait ignoring covariate effects of age and sex leads to overestimated power and underestimated replication sample size.

stat.ME↗

arXiv2022

Direction and Trajectory Tracking Control for Nonholonomic Spherical Robot by Combining Sliding Mode Controller and Model Prediction Controller

Spherical robot is a nonlinear, nonholonomic and unstable system which increases the difficulty of the direction and trajectory tracking problem. In this study, we propose a new direction controller HTSMC, an instruction planning controller MPC, and a trajectory tracking framework MHH. The HTSMC is designed by integrating a fast terminal algorithm, a hierarchical method, the motion features of a spherical robot, and its dynamics. In addition, the new direction controller has an excellent control effect with a quick response speed and strong stability. MPC can obtain optimal commands that are then transmitted to the velocity and direction controller. Since the two torque controllers in MHH are all Lyapunov-based sliding mode controllers, the MHH framework may achieve optimal control performance while assuring stability. Finally, the two controllers eliminate the requirement for MPC's stability and dynamic constraints. Finally, hardware experiments demonstrate the efficacy of the HTSMC, MPC, and MHH.

cs.RO↗

arXiv2019

Novel Defense Strategy Against Zero-Dynamics Attack in Multi-Agent Systems

This paper considers the defense strategy of strategic topology switching for the second-order multi-agent system under zero-dynamics attack (ZDA) whose attack-starting time is allowed to be not the initial time. We first study the detectability of ZDA, which is a sufficient and necessary condition of switching topologies in detecting the attack. Based on the detectability, a Luenberger observer under switching topology is proposed to detect the stealthy attack. The primary advantages of the attack-detection algorithm are twofold: (i) in detecting ZDA, the algorithm allows the defender (system operator) to have no knowledge of the attack-starting time and the misbehaving agents (i.e., agents under attack); (ii) in tracking system in the absence of attacks, Luenberger observer has no constraint on the magnitudes of observer gains and the number of observed outputs, i.e., only one agent's observed output is sufficient. Simulations are provided to verify the effectiveness of the strategic topology-switching algorithm.

eess.SY↗

arXiv2019

Strategic Topology Switching for Security-Part II: Detection & Switching Topologies

This two-part paper considers strategic topology switching for security in the second-order multi-agent system. In Part II, we propose a strategy on switching topologies to detect zero-dynamics attack (ZDA), whose attack-starting time is allowed to be not the initial time. We first characterize the sufficient and necessary condition for detectability of ZDA, in terms of the network topologies to be switched to and the set of agents to be monitored. We then propose an attack detection algorithm based on the Luenberger observer, using the characterized detectability condition. Employing the strategy on switching times proposed in Part I and the strategy on switching topologies proposed here, a strategic topology-switching algorithm is derived. Its primary advantages are threefold: (i) in achieving consensus in the absence of attacks, the control protocol does not need velocity measurements and the algorithm has no constraint on the magnitudes of coupling weights; (ii) in tracking system in the absence of attacks, the Luenberger observer has no constraint on the magnitudes of observer gains and the number of monitored agents, i.e., only one monitored agent's output is sufficient; (iii) in detecting ZDA, the algorithm allows the defender to have no knowledge of the attack-starting time and the number of misbehaving agents (i.e., agents under attack). Simulations are provided to verify the effectiveness of the strategic topology-switching algorithm.

cs.MA↗

arXiv2019

Strategic Topology Switching for Security-Part I: Consensus & Switching Times

In this two-part paper, we consider strategic topology switching for the second-order multi-agent systems under a special class of stealthy attacks, namely the "zero-dynamics" attack (ZDA). The main mathematical tool proposed here is to strategically switch the network topology to detect a possible ZDA. However, it is not clear a priori that such a switching strategy still yields consensus in this switched system, in the normal (un-attacked) operation mode. In Part I, we propose a strategy on the switching times that enables the topology-switching algorithm proposed in Part II to reach the second-order consensus in the absence of a ZDA. Utilizing the theory of stable switched linear systems with unstable subsystems, we characterize sufficient conditions for the dwell time of topology-switching signal to reach consensus. Building on this characterization, we then propose a decentralized time-dependent topology-switching algorithm. The proposed algorithm, used in conjunction with a simplified control protocol, achieves consensus while providing substantial advantages over other control approaches: it relies only on the relative position measurements (without any requirement for velocity measurements); and it does not impose any constraint on the magnitudes of coupling weights. We finally demonstrate our theoretical findings via the numerical simulation results.

eess.SY↗