arXiv subjects

Fei Zhang

Publications and source records attributed to Fei Zhang.

At least 37 records · Page 2Linked to original sources

arXiv2025

Decouple before Align: Visual Disentanglement Enhances Prompt Tuning

Prompt tuning (PT), as an emerging resource-efficient fine-tuning paradigm, has showcased remarkable effectiveness in improving the task-specific transferability of vision-language models. This paper delves into a previously overlooked information asymmetry issue in PT, where the visual modality mostly conveys more context than the object-oriented textual modality. Correspondingly, coarsely aligning these two modalities could result in the biased attention, driving the model to merely focus on the context area. To address this, we propose DAPT, an effective PT framework based on an intuitive decouple-before-align concept. First, we propose to explicitly decouple the visual modality into the foreground and background representation via exploiting coarse-and-fine visual segmenting cues, and then both of these decoupled patterns are aligned with the original foreground texts and the hand-crafted background classes, thereby symmetrically strengthening the modal alignment. To further enhance the visual concentration, we propose a visual pull-push regularization tailored for the foreground-background patterns, directing the original visual representation towards unbiased attention on the region-of-interest object. We demonstrate the power of architecture-free DAPT through few-shot learning, base-to-novel generalization, and data-efficient learning, all of which yield superior performance across prevailing benchmarks. Our code will be released at https://github.com/Ferenas/DAPT.

cs.CV↗

arXiv2025

ConText: Driving In-context Learning for Text Removal and Segmentation

This paper presents the first study on adapting the visual in-context learning (V-ICL) paradigm to optical character recognition tasks, specifically focusing on text removal and segmentation. Most existing V-ICL generalists employ a reasoning-as-reconstruction approach: they turn to using a straightforward image-label compositor as the prompt and query input, and then masking the query label to generate the desired output. This direct prompt confines the model to a challenging single-step reasoning process. To address this, we propose a task-chaining compositor in the form of image-removal-segmentation, providing an enhanced prompt that elicits reasoning with enriched intermediates. Additionally, we introduce context-aware aggregation, integrating the chained prompt pattern into the latent query representation, thereby strengthening the model's in-context reasoning. We also consider the issue of visual heterogeneity, which complicates the selection of homogeneous demonstrations in text recognition. Accordingly, this is effectively addressed through a simple self-prompting strategy, preventing the model's in-context learnability from devolving into specialist-like, context-free inference. Collectively, these insights culminate in our ConText model, which achieves new state-of-the-art across both in- and out-of-domain benchmarks. The code is available at https://github.com/Ferenas/ConText.

cs.CV↗

arXiv2025

G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Models

This paper considers the problem of utilizing a large-scale text-to-image diffusion model to tackle the challenging Inexact Segmentation (IS) task. Unlike traditional approaches that rely heavily on discriminative-model-based paradigms or dense visual representations derived from internal attention mechanisms, our method focuses on the intrinsic generative priors in Stable Diffusion~(SD). Specifically, we exploit the pattern discrepancies between original images and mask-conditional generated images to facilitate a coarse-to-fine segmentation refinement by establishing a semantic correspondence alignment and updating the foreground probability. Comprehensive quantitative and qualitative experiments validate the effectiveness and superiority of our plug-and-play design, underscoring the potential of leveraging generation discrepancies to model dense representations and encouraging further exploration of generative approaches for solving discriminative tasks.

cs.CV↗

arXiv2024

Pressure induced superconducting dome in LaNiGa2

LaNiGa2 is a time-reversal symmetry breaking superconductor with symmetry protected band crossings, making it an ideal platform for investigating the interplay between unconventional superconductivity and electronic structure topology. Here we present a transport study of LaNiGa2 under pressure. The application of pressure to LaNiGa2 induces a significant enhancement of the superconducting transition temperature Tc at a pressure of 7 GPa. In contrast, powder X-ray diffraction (XRD) results show no evidence of structural phase transitions up to 26.3 GPa. Moreover, the ratio of band diffusivity shows a sudden increase at around 7 GPa, suggesting possible pressure-induced changes in the electronic structure that are closely linked to the evolution of superconductivity.

cond-mat.supr-con↗

arXiv2024

Image Segmentation in Foundation Model Era: A Survey

Image segmentation is a long-standing challenge in computer vision, studied continuously over several decades, as evidenced by seminal algorithms such as N-Cut, FCN, and MaskFormer. With the advent of foundation models (FMs), contemporary segmentation methodologies have embarked on a new epoch by either adapting FMs (e.g., CLIP, Stable Diffusion, DINO) for image segmentation or developing dedicated segmentation foundation models (e.g., SAM). These approaches not only deliver superior segmentation performance, but also herald newfound segmentation capabilities previously unseen in deep learning context. However, current research in image segmentation lacks a detailed analysis of distinct characteristics, challenges, and solutions associated with these advancements. This survey seeks to fill this gap by providing a thorough review of cutting-edge research centered around FM-driven image segmentation. We investigate two basic lines of research -- generic image segmentation (i.e., semantic segmentation, instance segmentation, panoptic segmentation), and promptable image segmentation (i.e., interactive segmentation, referring segmentation, few-shot segmentation) -- by delineating their respective task settings, background concepts, and key challenges. Furthermore, we provide insights into the emergence of segmentation knowledge from FMs like CLIP, Stable Diffusion, and DINO. An exhaustive overview of over 300 segmentation approaches is provided to encapsulate the breadth of current research efforts. Subsequently, we engage in a discussion of open issues and potential avenues for future research. We envisage that this fresh, comprehensive, and systematic survey catalyzes the evolution of advanced image segmentation systems. A public website is created to continuously track developments in this fast advancing field: \url{https://github.com/stanley-313/ImageSegFM-Survey}.

cs.CV↗

arXiv2024

200mm Optical synthetic aperture imaging over 120 meters distance via Macroscopic Fourier ptychography

Fourier ptychography (FP) imaging, drawing on the idea of synthetic aperture, has been demonstrated as a potential approach for remote sub-diffraction-limited imaging. Nevertheless, the farthest imaging distance is still limited around 10 m even though there has been a significant improvement in macroscopic FP. The most severely issue in increasing the imaging distance is field of view (FoV) limitation caused by far-field condition for diffraction. Here, we propose to modify the Fourier far-field condition for rough reflective objects, aiming to overcome the small FoV limitation by using a divergent beam to illuminate objects. A joint optimization of pupil function and target image is utilized to attain the aberration-free image while estimating the pupil function simultaneously. Benefiting from the optimized reconstruction algorithm which effectively expands the camera's effective aperture, we experimentally implement several FP systems suited for imaging distance of 12 m, 65 m and 120m with the maximum synthetic aperture of 200 mm. The maximum synthetic aperture is thus improved by more than one order of magnitude of the state-of-the-art works from the furthest distance, with an over fourfold improvement in the resolution compare to single aperture. Our findings demonstrate significant potential for advancing the field of macroscopic FP, propelling it into a new stage of development.

physics.optics↗

arXiv2024

Multi-label Classification for Android Malware Based on Active Learning

The existing malware classification approaches (i.e., binary and family classification) can barely benefit subsequent analysis with their outputs. Even the family classification approaches suffer from lacking a formal naming standard and an incomplete definition of malicious behaviors. More importantly, the existing approaches are powerless for one malware with multiple malicious behaviors, while this is a very common phenomenon for Android malware in the wild. So, neither of them can provide researchers with a direct and comprehensive enough understanding of malware. In this paper, we propose MLCDroid, an ML-based multi-label classification approach that can directly indicate the existence of pre-defined malicious behaviors. With an in-depth analysis, we summarize six basic malicious behaviors from real-world malware with security reports and construct a labeled dataset. We compare the results of 70 algorithm combinations to evaluate the effectiveness (best at 73.3%). Faced with the challenge of the expensive cost of data annotation, we further propose an active learning approach based on data augmentation, which can improve the overall accuracy to 86.7% with a data augmentation of 5,000+ high-quality samples from an unlabeled malware dataset. This is the first multi-label Android malware classification approach intending to provide more information on fine-grained malicious behaviors.

cs.CR↗

arXiv2024

General Impedance Modeling for Modular Multilevel Converter with Grid-forming and Grid-following Control

Modular multilevel converter (MMC) has complex topology, control architecture and broadband harmonic spectrum. For this, linear-time-periodic (LTP) theory, covering multi-harmonic coupling relations, has been adopted for MMC impedance modeling recently. However, the existing MMC impedance models usually lack explicit expressions and general modeling procedure for different control strategies. To this end, this paper proposes a general impedance modeling procedure applicable to various power converters with grid-forming and grid-following control strategies. The modeling is based on a unified representation of MMC circuit as the input and output relation between the voltage or current on the AC side and the exerted modulation index, while the control part vice versa, thereby interconnected as closed-loop feedback. With each part expressed as transfer functions, the final impedance model keeps the explicit form of harmonic transfer function matrix, making it convenient to directly observe and analyze the influence of each part individually. Thereby the submodule capacitance is found as the main cause of difference between MMC impedance compared to two-level converter, which will get closer as the capacitance increases. Effectiveness and generality of the impedance modeling method is demonstrated through comprehensive comparison with impedance scanning using electromagnetic transient simulation.

eess.SY↗

arXiv2024

Learning the expressibility of quantum circuit ansatz using transformer

With the exponentially faster computation for certain problems, quantum computing has garnered significant attention in recent years. Variational quantum algorithms are crucial methods to implement quantum computing, and an appropriate task-specific quantum circuit ansatz can effectively enhance the quantum advantage of VQAs. However, the vast search space makes it challenging to find the optimal task-specific ansatz. Expressibility, quantifying the diversity of quantum circuit ansatz states to explore the Hilbert space effectively, can be used to evaluate whether one ansatz is superior to another. In this work, we propose using a transformer model to predict the expressibility of quantum circuit ansatze. We construct a dataset containing random PQCs generated by the gatewise pipeline, with varying numbers of qubits and gates. The expressibility of the circuits is calculated using three measures: KL divergence, relative KL divergence, and maximum mean discrepancy. A transformer model is trained on the dataset to capture the intricate relationships between circuit characteristics and expressibility. Four evaluation metrics are employed to assess the performance of the transformer. Numerical results demonstrate that the trained model achieves high performance and robustness across various expressibility measures. This research can enhance the understanding of the expressibility of quantum circuit ansatze and advance quantum architecture search algorithms.

quant-ph↗

arXiv2024

Multi-Modal Prototypes for Open-World Semantic Segmentation

In semantic segmentation, generalizing a visual system to both seen categories and novel categories at inference time has always been practically valuable yet challenging. To enable such functionality, existing methods mainly rely on either providing several support demonstrations from the visual aspect or characterizing the informative clues from the textual aspect (e.g., the class names). Nevertheless, both two lines neglect the complementary intrinsic of low-level visual and high-level language information, while the explorations that consider visual and textual modalities as a whole to promote predictions are still limited. To close this gap, we propose to encompass textual and visual clues as multi-modal prototypes to allow more comprehensive support for open-world semantic segmentation, and build a novel prototype-based segmentation framework to realize this promise. To be specific, unlike the straightforward combination of bi-modal clues, we decompose the high-level language information as multi-aspect prototypes and aggregate the low-level visual information as more semantic prototypes, on basis of which, a fine-grained complementary fusion makes the multi-modal prototypes more powerful and accurate to promote the prediction. Based on an elastic mask prediction module that permits any number and form of prototype inputs, we are able to solve the zero-shot, few-shot and generalized counterpart tasks in one architecture. Extensive experiments on both PASCAL-$5^i$ and COCO-$20^i$ datasets show the consistent superiority of the proposed method compared with the previous state-of-the-art approaches, and a range of ablation studies thoroughly dissects each component in our framework both quantitatively and qualitatively that verify their effectiveness.

cs.CV↗

arXiv2024

Audio-Visual Segmentation via Unlabeled Frame Exploitation

Audio-visual segmentation (AVS) aims to segment the sounding objects in video frames. Although great progress has been witnessed, we experimentally reveal that current methods reach marginal performance gain within the use of the unlabeled frames, leading to the underutilization issue. To fully explore the potential of the unlabeled frames for AVS, we explicitly divide them into two categories based on their temporal characteristics, i.e., neighboring frame (NF) and distant frame (DF). NFs, temporally adjacent to the labeled frame, often contain rich motion information that assists in the accurate localization of sounding objects. Contrary to NFs, DFs have long temporal distances from the labeled frame, which share semantic-similar objects with appearance variations. Considering their unique characteristics, we propose a versatile framework that effectively leverages them to tackle AVS. Specifically, for NFs, we exploit the motion cues as the dynamic guidance to improve the objectness localization. Besides, we exploit the semantic cues in DFs by treating them as valid augmentations to the labeled frames, which are then used to enrich data diversity in a self-training manner. Extensive experimental results demonstrate the versatility and superiority of our method, unleashing the power of the abundant unlabeled frames.

cs.CV↗

arXiv2024

AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation

Open-vocabulary semantic segmentation is a challenging task that requires segmenting novel object categories at inference time. Recent studies have explored vision-language pre-training to handle this task, but suffer from unrealistic assumptions in practical scenarios, i.e., low-quality textual category names. For example, this paradigm assumes that new textual categories will be accurately and completely provided, and exist in lexicons during pre-training. However, exceptions often happen when encountering ambiguity for brief or incomplete names, new words that are not present in the pre-trained lexicons, and difficult-to-describe categories for users. To address these issues, this work proposes a novel attribute decomposition-aggregation framework, AttrSeg, inspired by human cognition in understanding new concepts. Specifically, in the decomposition stage, we decouple class names into diverse attribute descriptions to complement semantic contexts from multiple perspectives. Two attribute construction strategies are designed: using large language models for common categories, and involving manually labeling for human-invented categories. In the aggregation stage, we group diverse attributes into an integrated global description, to form a discriminative classifier that distinguishes the target object from others. One hierarchical aggregation architecture is further proposed to achieve multi-level aggregations, leveraging the meticulously designed clustering module. The final results are obtained by computing the similarity between aggregated attributes and images embeddings. To evaluate the effectiveness, we annotate three types of datasets with attribute descriptions, and conduct extensive experiments and ablation studies. The results show the superior performance of attribute decomposition-aggregation.

cs.CV↗

arXiv2023

Long-term and Real-time High-speed Underwater Wireless Optical Communications in Deep Sea

Seafloor observation network can perform all-weather, long-term, continuous, real-time, and in-situ observation of the ocean by combing various observation methods including cabled seafloor nodes, self-contained nodes, as well as mobile platforms, where reliable and long-term high-speed underwater wireless communication becomes an essential demand. Recently, underwater wireless optical communication (UWOC) has emerged as a highly promising solution and is rapidly becoming a research hotspot for meeting this requirement. In this article, we demonstrate the experiment and application of high-speed UWOC system for deep sea seafloor observation network. To the best of our knowledge this is the first long-term real-time deep-sea UWOC link with bitrate as high as 125 Mbps. Between 30 m distance and at a depth of 1650 m, two-way Ethernet UWOC links are realized with 125 Mbps direction-adjustable green light link and 6.25 Mbps non-line-of-sight (NLOS) blue light link. High quality video transmission of 8K 30 FPS and 4K 120 FPS are realized through high-speed 125 Mbps green light link, with 100% peak signal-to-noise ratio (PSNR) agreement, showing the capability of transmitting high-quality videos lossless. The 30-day long-term measurement results show that the BER performance of both 125 Mbps and 6.25 Mbps links is lower than 10-5, proving the stability and reliability of this UWOC system at depth of 1650 m. The maximum transmission distance for the green and blue light links are estimated to be 117.7 and 128.3 m with considering the geometry loss, which can be extended to 231.6 and 337.5 m without geometry loss. As the first long-term and real-time UWOC system in deep sea, we believe this demonstration can provide valuable experience for further UWOC studies and converged ocean observation networking with cabled and cable-less observation platforms.

cs.NI↗

arXiv2023

Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation

This paper studies the problem of weakly open-vocabulary semantic segmentation (WOVSS), which learns to segment objects of arbitrary classes using mere image-text pairs. Existing works turn to enhance the vanilla vision transformer by introducing explicit grouping recognition, i.e., employing several group tokens/centroids to cluster the image tokens and perform the group-text alignment. Nevertheless, these methods suffer from a granularity inconsistency regarding the usage of group tokens, which are aligned in the all-to-one v.s. one-to-one manners during the training and inference phases, respectively. We argue that this discrepancy arises from the lack of elaborate supervision for each group token. To bridge this granularity gap, this paper explores explicit supervision for the group tokens from the prototypical knowledge. To this end, this paper proposes the non-learnable prototypical regularization (NPR) where non-learnable prototypes are estimated from source features to serve as supervision and enable contrastive matching of the group tokens. This regularization encourages the group tokens to segment objects with less redundancy and capture more comprehensive semantic regions, leading to increased compactness and richness. Based on NPR, we propose the prototypical guidance segmentation network (PGSeg) that incorporates multi-modal regularization by leveraging prototypical sources from both images and texts at different levels, progressively enhancing the segmentation capability with diverse prototypical patterns. Experimental results show that our proposed method achieves state-of-the-art performance on several benchmark datasets. The source code is available at https://github.com/Ferenas/PGSeg.

cs.CV↗

arXiv2023

Exploiting Counter-Examples for Active Learning with Partial labels

This paper studies a new problem, \emph{active learning with partial labels} (ALPL). In this setting, an oracle annotates the query samples with partial labels, relaxing the oracle from the demanding accurate labeling process. To address ALPL, we first build an intuitive baseline that can be seamlessly incorporated into existing AL frameworks. Though effective, this baseline is still susceptible to the \emph{overfitting}, and falls short of the representative partial-label-based samples during the query process. Drawing inspiration from human inference in cognitive science, where accurate inferences can be explicitly derived from \emph{counter-examples} (CEs), our objective is to leverage this human-like learning pattern to tackle the \emph{overfitting} while enhancing the process of selecting representative samples in ALPL. Specifically, we construct CEs by reversing the partial labels for each instance, and then we propose a simple but effective WorseNet to directly learn from this complementary pattern. By leveraging the distribution gap between WorseNet and the predictor, this adversarial evaluation manner could enhance both the performance of the predictor itself and the sample selection process, allowing the predictor to capture more accurate patterns in the data. Experimental results on five real-world datasets and four benchmark datasets show that our proposed method achieves comprehensive improvements over ten representative AL frameworks, highlighting the superiority of WorseNet. The source code will be available at \url{https://github.com/Ferenas/APLL}.

cs.LG↗

arXiv2023

Prediction of Post-Operative Renal and Pulmonary Complications Using Transformers

Postoperative complications pose a significant challenge in the healthcare industry, resulting in elevated healthcare expenses and prolonged hospital stays, and in rare instances, patient mortality. To improve patient outcomes and reduce healthcare costs, healthcare providers rely on various perioperative risk scores to guide clinical decisions and prioritize care. In recent years, machine learning techniques have shown promise in predicting postoperative complications and fatality, with deep learning models achieving remarkable success in healthcare applications. However, research on the application of deep learning models to intra-operative anesthesia management data is limited. In this paper, we evaluate the performance of transformer-based models in predicting postoperative acute renal failure, postoperative pulmonary complications, and postoperative in-hospital mortality. We compare our method's performance with state-of-the-art tabular data prediction models, including gradient boosting trees and sequential attention models, on a clinical dataset. Our results demonstrate that transformer-based models can achieve superior performance in predicting postoperative complications and outperform traditional machine learning models. This work highlights the potential of deep learning techniques, specifically transformer-based models, in revolutionizing the healthcare industry's approach to postoperative care.

cs.LG↗

arXiv2023

Physics-data-driven intelligent optimization for large-scale meta-devices

Meta-devices have gained significant attention and have been widely utilized in optical systems for focusing and imaging, owing to their lightweight, high-integration, and exceptional-flexibility capabilities. However, based on the assumption of local phase approximation, traditional design method neglect the local lattice coupling effect between adjacent meta-atoms, thus harming the practical performance of meta-devices. Using physics-driven or data-driven optimization algorithms can effectively solve the aforementioned problems. Nevertheless, both of the methods either involve considerable time costs or require a substantial amount of data sets. Here, we propose a physics-data-driven approach based "intelligent optimizer" that enables us to adaptively modify the sizes of the studied meta-atom according to the sizes of its surrounding ones. Such a scheme allows to mitigate the undesired local lattice coupling effect, and the proposed network model works well on thousands of datasets with a validation loss of 3*10-3. Experimental results show that the 1-mm-diameter metalens designed with the "intelligent optimizer" possesses a relative focusing efficiency of 93.4% (as compared to ideal focusing) and a Strehl ratio of 0.94. In contrast to the previous inverse design method, our method significantly boosts designing efficiency with five orders of magnitude reduction in time. Our design approach may sets a new paradigm for devising large-scale meta-devices.

physics.optics↗

arXiv2023

Monte Carlo Linear Clustering with Single-Point Supervision is Enough for Infrared Small Target Detection

Single-frame infrared small target (SIRST) detection aims at separating small targets from clutter backgrounds on infrared images. Recently, deep learning based methods have achieved promising performance on SIRST detection, but at the cost of a large amount of training data with expensive pixel-level annotations. To reduce the annotation burden, we propose the first method to achieve SIRST detection with single-point supervision. The core idea of this work is to recover the per-pixel mask of each target from the given single point label by using clustering approaches, which looks simple but is indeed challenging since targets are always insalient and accompanied with background clutters. To handle this issue, we introduce randomness to the clustering process by adding noise to the input images, and then obtain much more reliable pseudo masks by averaging the clustered results. Thanks to this "Monte Carlo" clustering approach, our method can accurately recover pseudo masks and thus turn arbitrary fully supervised SIRST detection networks into weakly supervised ones with only single point annotation. Experiments on four datasets demonstrate that our method can be applied to existing SIRST detection networks to achieve comparable performance with their fully supervised counterparts, which reveals that single-point supervision is strong enough for SIRST detection. Our code will be available at: https://github.com/YeRen123455/SIRST-Single-Point-Supervision.

cs.CV↗