arXiv subjects

Jing Zhang

Publications and source records attributed to Jing Zhang.

At least 397 records · Page 22Linked to original sources

arXiv2024

Patch-level Sounding Object Tracking for Audio-Visual Question Answering

Answering questions related to audio-visual scenes, i.e., the AVQA task, is becoming increasingly popular. A critical challenge is accurately identifying and tracking sounding objects related to the question along the timeline. In this paper, we present a new Patch-level Sounding Object Tracking (PSOT) method. It begins with a Motion-driven Key Patch Tracking (M-KPT) module, which relies on visual motion information to identify salient visual patches with significant movements that are more likely to relate to sounding objects and questions. We measure the patch-wise motion intensity map between neighboring video frames and utilize it to construct and guide a motion-driven graph network. Meanwhile, we design a Sound-driven KPT (S-KPT) module to explicitly track sounding patches. This module also involves a graph network, with the adjacency matrix regularized by the audio-visual correspondence map. The M-KPT and S-KPT modules are performed in parallel for each temporal segment, allowing balanced tracking of salient and sounding objects. Based on the tracked patches, we further propose a Question-driven KPT (Q-KPT) module to retain patches highly relevant to the question, ensuring the model focuses on the most informative clues. The audio-visual-question features are updated during the processing of these modules, which are then aggregated for final answer prediction. Extensive experiments on standard datasets demonstrate the effectiveness of our method, achieving competitive performance even compared to recent large-scale pretraining-based approaches.

cs.MM↗

arXiv2024

SVGDreamer++: Advancing Editability and Diversity in Text-Guided SVG Generation

Recently, text-guided scalable vector graphics (SVG) synthesis has demonstrated significant potential in domains such as iconography and sketching. However, SVGs generated from existing Text-to-SVG methods often lack editability and exhibit deficiencies in visual quality and diversity. In this paper, we propose a novel text-guided vector graphics synthesis method to address these limitations. To enhance the editability of output SVGs, we introduce a Hierarchical Image VEctorization (HIVE) framework that operates at the semantic object level and supervises the optimization of components within the vector object. This approach facilitates the decoupling of vector graphics into distinct objects and component levels. Our proposed HIVE algorithm, informed by image segmentation priors, not only ensures a more precise representation of vector graphics but also enables fine-grained editing capabilities within vector objects. To improve the diversity of output SVGs, we present a Vectorized Particle-based Score Distillation (VPSD) approach. VPSD addresses over-saturation issues in existing methods and enhances sample diversity. A pre-trained reward model is incorporated to re-weight vector particles, improving aesthetic appeal and enabling faster convergence. Additionally, we design a novel adaptive vector primitives control strategy, which allows for the dynamic adjustment of the number of primitives, thereby enhancing the presentation of graphic details. Extensive experiments validate the effectiveness of the proposed method, demonstrating its superiority over baseline methods in terms of editability, visual quality, and diversity. We also show that our new method supports up to six distinct vector styles, capable of generating high-quality vector assets suitable for stylized vector design and poster design. Code and demo will be released at: http://ximinng.github.io/SVGDreamerV2Project/

cs.CV↗

arXiv2024

Near-Optimal MIMO Detection Using Gradient-Based MCMC in Discrete Spaces

The discrete nature of transmitted symbols poses challenges for achieving optimal detection in multiple-input multiple-output (MIMO) systems associated with a large number of antennas. Recently, the combination of two powerful machine learning methods, Markov chain Monte Carlo (MCMC) sampling and gradient descent, has emerged as a highly efficient solution to address this issue. However, existing gradient-based MCMC detectors are heuristically designed and thus are theoretically untenable. To bridge this gap, we introduce a novel sampling algorithm tailored for discrete spaces. This algorithm leverages gradients from the underlying continuous spaces for acceleration while maintaining the validity of probabilistic sampling. We prove the convergence of this method and also analyze its convergence rate using both MCMC theory and empirical diagnostics. On this basis, we develop a MIMO detector that precisely samples from the target discrete distribution and generates posterior Bayesian estimates using these samples, whose performance is thereby theoretically guaranteed. Furthermore, our proposed detector is highly parallelizable and scalable to large MIMO dimensions, positioning it as a compelling candidate for next-generation wireless networks. Simulation results show that our detector achieves near-optimal performance, significantly outperforms state-of-the-art baselines, and showcases resilience to various system setups.

eess.SP↗

arXiv2024

A Generative Victim Model for Segmentation

We find that the well-trained victim models (VMs), against which the attacks are generated, serve as fundamental prerequisites for adversarial attacks, i.e. a segmentation VM is needed to generate attacks for segmentation. In this context, the victim model is assumed to be robust to achieve effective adversarial perturbation generation. Instead of focusing on improving the robustness of the task-specific victim models, we shift our attention to image generation. From an image generation perspective, we derive a novel VM for segmentation, aiming to generate adversarial perturbations for segmentation tasks without requiring models explicitly designed for image segmentation. Our approach to adversarial attack generation diverges from conventional white-box or black-box attacks, offering a fresh outlook on adversarial attack strategies. Experiments show that our attack method is able to generate effective adversarial attacks with good transferability.

cs.CV↗

arXiv2024

Precision Enhancement in Spatial Measurement by Introducing Squeezed Light into Weak Value Amplification

The precision enhancement is demonstrated in an optical spatial measurement based on weak value amplification (WVA) system and split-like detection, by injecting a TEM$_{10}$ squeezed vacuum beam. It is the first time to experimentally realize high-precision optical spatial measurement beyond the shot noise limit by using squeezing-assisted WVA. Based on the WVA technique, which can amplify the signal by increasing the number of photons that injected into the interferometer, squeezed beam injection can reduce the noise level and can further improve the signal-to-noise ratio(SNR). As a result, a SNR improvement of 2dB, i.e., 1.3 time precision enhancement, can be achieved, by using a 2dB squeezed beam injection and 2.6\% postselection probability in WVA process. Our work provides an effective method to accomplish higher precision in quantum spatial measurement, which has potential applications in gravitational wave interferometer calibration, super-resolution quantum imaging, etc.

quant-ph↗

arXiv2024

Responsivity evaluation of photonics integrated photodetectors via pairwise measurements with an attenuation circuit

Integrated photonics platforms offer a compact and scalable solution for developing next-generation optical technologies. For precision applications involving weak signals, the responsivity as well as the accurate calibration of the integrated photodetectors at low optical powers become increasingly important. It remains challenging to perform a calibration traceable to mW-level primary standards without relying on external attenuation setups. Here, we utilize an on-chip attenuation circuit, composed of a series of cascaded directional couplers (DCs), to evaluate the responsivity of integrated photodetectors (PDs) at uW optical power levels with mW inputs to the chip. Moreover, we show that a pairwise measurement method, involving the simultaneous measurement of the integrated PD photocurrent and an auxiliary optical output which is coupled off-chip, systematically improves the experimental uncertainties compared to a direct PD photocurrent measurement. For 3 cascaded DCs, the pairwise measurement improves the repeatability error from 1.21% to 0.22%, with an overall expanded calibration uncertainty (k=2) of 10.13%. The latter is dominated by the scattering noise floor and fiber-to-chip coupling errors, which can be significantly improved with better device fabrication control. Our method can be extended to a fully integrated calibration solution for waveguide-integrated single-photon detectors.

physics.optics↗

arXiv2024

7 Tesla multimodal MRI dataset of ex-vivo human brain

Ex-vivo MRI offers invaluable insights into the complexity of the human brain, enabling high-resolution anatomical delineation and integration with histopathology, and thus, contributes to both basic and clinical studies on normal and pathological brains. However, ex-vivo MRI is challenging in sample preparation, acquisition, and data analysis, and existing ex-vivo MRI datasets are often single image modality and lack of ethnic diversity. In our study, we aimed to address these limitations by constructing a comprehensive multimodal MRI database acquired from six ex-vivo Chinese human brains. This database included structural MRI, high-angular resolution diffusion MRI, quantitative susceptibility mapping, and quantitative T1 and T2 maps, which enabled multifaceted depiction of brain microstructure and connectivity. Furthermore, we generated population-averaged multimodal templates and the segmentation labels to facilitate analysis of ex-vivo brain MRI. This public database offers a collection of high-resolution and multi-parametric ex-vivo human brain MRI and filled the gap of lacking Asian brain samples in existing databases.

q-bio.QM↗

arXiv2024

Quality Control in Open-Ended Crowdsourcing: A Survey

Crowdsourcing provides a flexible approach for leveraging human intelligence to solve large-scale problems, gaining widespread acceptance in domains like intelligent information processing, social decision-making, and crowd ideation. However, the uncertainty of participants significantly compromises the answer quality, sparking substantial research interest. Existing surveys predominantly concentrate on quality control in Boolean tasks, which are generally formulated as simple label classification, ranking, or numerical prediction. Ubiquitous open-ended tasks like question-answering, translation, and semantic segmentation have not been sufficiently discussed. These tasks usually have large to infinite answer spaces and non-unique acceptable answers, posing significant challenges for quality assurance. This survey focuses on quality control methods applicable to open-ended tasks in crowdsourcing. We propose a two-tiered framework to categorize related works. The first tier introduces a holistic view of the quality model, encompassing key aspects like task, worker, answer, and system. The second tier refines the classification into more detailed categories, including quality dimensions, evaluation metrics, and design decisions, providing insights into the internal structures of the quality control framework in each aspect. We thoroughly investigate how these quality control methods are implemented in state-of-the-art works and discuss key challenges and potential future research directions.

cs.HC↗

arXiv2024

Quantized Landau-level crossing checkerboard in large-angle twisted graphene

When charge transport occurs under conditions like topological protection or ballistic motion, the conductance of low-dimensional systems often exhibits quantized values in units of $e^{2}/h$, where $e$ and $h$ are the elementary charge and Planck's constant. Such quantization has been pivotal in quantum metrology and computing. Here, we demonstrate a novel quantized quantity: the ratio of the displacement field to the magnetic field, $D/B$, in large-twist-angle bilayer graphene. In the high magnetic field limit, Landau level crossings between the top and bottom layers manifest equal-sized checkerboard patterns throughout the $D/B$-$ν$ space. It stems from a peculiar electric-field-driven interlayer charge transfer at one elementary charge per flux quantum, leading to quantized intervals of critical displacement fields, (i.e., $δD$ = $\frac{e}{2πl_{B}^{2}}$, where $l_B$ is the magnetic length). Our findings suggest that interlayer charge transfer in the quantum Hall regime can yield intriguing physical phenomena, which has been overlooked in the past.

cond-mat.mes-hall↗

arXiv2024

Analyzing Nobel Prize Literature with Large Language Models

This study examines the capabilities of advanced Large Language Models (LLMs), particularly the o1 model, in the context of literary analysis. The outputs of these models are compared directly to those produced by graduate-level human participants. By focusing on two Nobel Prize-winning short stories, 'Nine Chapters' by Han Kang, the 2024 laureate, and 'Friendship' by Jon Fosse, the 2023 laureate, the research explores the extent to which AI can engage with complex literary elements such as thematic analysis, intertextuality, cultural and historical contexts, linguistic and structural innovations, and character development. Given the Nobel Prize's prestige and its emphasis on cultural, historical, and linguistic richness, applying LLMs to these works provides a deeper understanding of both human and AI approaches to interpretation. The study uses qualitative and quantitative evaluations of coherence, creativity, and fidelity to the text, revealing the strengths and limitations of AI in tasks typically reserved for human expertise. While LLMs demonstrate strong analytical capabilities, particularly in structured tasks, they often fall short in emotional nuance and coherence, areas where human interpretation excels. This research underscores the potential for human-AI collaboration in the humanities, opening new opportunities in literary studies and beyond.

cs.CL↗

arXiv2024

Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data

Over the past decade, the revolution in single-cell sequencing has enabled the simultaneous molecular profiling of various modalities across thousands of individual cells, allowing scientists to investigate the diverse functions of complex tissues and uncover underlying disease mechanisms. Among all the analytical steps, assigning individual cells to specific types is fundamental for understanding cellular heterogeneity. However, this process is usually labor-intensive and requires extensive expert knowledge. Recent advances in large language models (LLMs) have demonstrated their ability to efficiently process and synthesize vast corpora of text to automatically extract essential biological knowledge, such as marker genes, potentially promoting more efficient and automated cell type annotations. To thoroughly evaluate the capability of modern instruction-tuned LLMs in automating the cell type identification process, we introduce SOAR, a comprehensive benchmarking study of LLMs for cell type annotation tasks in single-cell genomics. Specifically, we assess the performance of 8 instruction-tuned LLMs across 11 datasets, spanning multiple cell types and species. Our study explores the potential of LLMs to accurately classify and annotate cell types in single-cell RNA sequencing (scRNA-seq) data, while extending their application to multiomics data through cross-modality translation. Additionally, we evaluate the effectiveness of chain-of-thought (CoT) prompting techniques in generating detailed biological insights during the annotation process. The results demonstrate that LLMs can provide robust interpretations of single-cell data without requiring additional fine-tuning, advancing the automation of cell type annotation in genomics research.

cs.CL↗

arXiv2024

How Texts Help? A Fine-grained Evaluation to Reveal the Role of Language in Vision-Language Tracking

Vision-language tracking (VLT) extends traditional single object tracking by incorporating textual information, providing semantic guidance to enhance tracking performance under challenging conditions like fast motion and deformations. However, current VLT trackers often underperform compared to single-modality methods on multiple benchmarks, with semantic information sometimes becoming a "distraction." To address this, we propose VLTVerse, the first fine-grained evaluation framework for VLT trackers that comprehensively considers multiple challenge factors and diverse semantic information, hoping to reveal the role of language in VLT. Our contributions include: (1) VLTVerse introduces 10 sequence-level challenge labels and 6 types of multi-granularity semantic information, creating a flexible and multi-dimensional evaluation space for VLT; (2) leveraging 60 subspaces formed by combinations of challenge factors and semantic types, we conduct systematic fine-grained evaluations of three mainstream SOTA VLT trackers, uncovering their performance bottlenecks across complex scenarios and offering a novel perspective on VLT evaluation; (3) through decoupled analysis of experimental results, we examine the impact of various semantic types on specific challenge factors in relation to different algorithms, providing essential guidance for enhancing VLT across data, evaluation, and algorithmic dimensions. The VLTVerse, toolkit, and results will be available at \url{http://metaverse.aitestunion.com}.

cs.CV↗

arXiv2024

AEAKA: An Adaptive and Efficient Authentication and Key Agreement Scheme for IoT in Cloud-Edge-Device Collaborative Environments

To meet the diverse needs of users, the rapid advancement of cloud-edge-device collaboration has become a standard practice. However, this complex environment, particularly in untrusted (non-collaborative) scenarios, presents numerous security challenges. Authentication acts as the first line of defense and is fundamental to addressing these issues. Although many authentication and key agreement schemes exist, they often face limitations, such as being tailored to overly specific scenarios where devices authenticate solely with either the edge or the cloud, or being unsuitable for resource-constrained devices. To address these challenges, we propose an adaptive and efficient authentication and key agreement scheme (AEAKA) for Cloud-Edge-Device IoT environments. This scheme is highly adaptive and scalable, capable of automatically and dynamically initiating different authentication methods based on device requirements. Additionally, it employs an edge-assisted authentication approach to reduce the load on third-party trust authorities. Furthermore, we introduce a hash-based algorithm for the authentication protocol, ensuring a lightweight method suitable for a wide range of resource-constrained devices while maintaining security. AEAKA ensures that entities use associated authentication credentials, enhancing the privacy of the authentication process. Security proofs and performance analyses demonstrate that AEAKA outperforms other methods in terms of security and authentication efficiency.

cs.CR↗

arXiv2024

UNSCT-HRNet: Modeling Anatomical Uncertainty for Landmark Detection in Total Hip Arthroplasty

Total hip arthroplasty (THA) relies on accurate landmark detection from radiographic images, but unstructured data caused by irregular patient postures or occluded anatomical markers pose significant challenges for existing methods. To address this, we propose UNSCT-HRNet (Unstructured CT - High-Resolution Net), a deep learning-based framework that integrates a Spatial Relationship Fusion (SRF) module and an Uncertainty Estimation (UE) module. The SRF module, utilizing coordinate convolution and polarized attention, enhances the model's ability to capture complex spatial relationships. Meanwhile, the UE module which based on entropy ensures predictions are anatomically relevant. For unstructured data, the proposed method can predict landmarks without relying on the fixed number of points, which shows higher accuracy and better robustness comparing with the existing methods. Our UNSCT-HRNet demonstrates over a 60% improvement across multiple metrics in unstructured data. The experimental results also reveal that our approach maintains good performance on the structured dataset. Overall, the proposed UNSCT-HRNet has the potential to be used as a new reliable, automated solution for THA surgical planning and postoperative monitoring.

eess.IV↗

arXiv2024

Effect of Top Al$_2$O$_3$ Interlayer Thickness on Memory Window and Reliability of FeFETs With TiN/Al$_2$O$_3$/Hf$_{0.5}$Zr$_{0.5}$O$_2$/SiO$_x$/Si (MIFIS) Gate Structure

We investigate the effect of top Al2O3 interlayer thickness on the memory window (MW) of Si channel ferroelectric field-effect transistors (Si-FeFETs) with TiN/Al$_2$O$_3$/Hf$_{0.5}$Zr$_{0.5}$O$_2$/SiO$_x$/Si (MIFIS) gate structure. We find that the MW first increases and then remains almost constant with the increasing thickness of the top Al2O3. The phenomenon is attributed to the lower electric field of the ferroelectric Hf$_{0.5}$Zr$_{0.5}$O$_2$ in the MIFIS structure with a thicker top Al2O3 after a program operation. The lower electric field makes the charges trapped at the top Al2O3/Hf0.5Zr0.5O$_2$ interface, which are injected from the metal gate, cannot be retained. Furthermore, we study the effect of the top Al$_2$O$_3$ interlayer thickness on the reliability (endurance characteristics and retention characteristics). We find that the MIFIS structure with a thicker top Al$_2$O$_3$ interlayer has poorer retention and endurance characteristics. Our work is helpful in deeply understanding the effect of top interlayer thickness on the MW and reliability of Si-FeFETs with MIFIS gate stacks.

cond-mat.mtrl-sci↗

arXiv2024

All-in-one Weather-degraded Image Restoration via Adaptive Degradation-aware Self-prompting Model

Existing approaches for all-in-one weather-degraded image restoration suffer from inefficiencies in leveraging degradation-aware priors, resulting in sub-optimal performance in adapting to different weather conditions. To this end, we develop an adaptive degradation-aware self-prompting model (ADSM) for all-in-one weather-degraded image restoration. Specifically, our model employs the contrastive language-image pre-training model (CLIP) to facilitate the training of our proposed latent prompt generators (LPGs), which represent three types of latent prompts to characterize the degradation type, degradation property and image caption. Moreover, we integrate the acquired degradation-aware prompts into the time embedding of diffusion model to improve degradation perception. Meanwhile, we employ the latent caption prompt to guide the reverse sampling process using the cross-attention mechanism, thereby guiding the accurate image reconstruction. Furthermore, to accelerate the reverse sampling procedure of diffusion model and address the limitations of frequency perception, we introduce a wavelet-oriented noise estimating network (WNE-Net). Extensive experiments conducted on eight publicly available datasets demonstrate the effectiveness of our proposed approach in both task-specific and all-in-one applications.

cs.CV↗

arXiv2024

Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentation

The Segment Anything Model (SAM), a profound vision foundation model pretrained on a large-scale dataset, breaks the boundaries of general segmentation and sparks various downstream applications. This paper introduces Hi-SAM, a unified model leveraging SAM for hierarchical text segmentation. Hi-SAM excels in segmentation across four hierarchies, including pixel-level text, word, text-line, and paragraph, while realizing layout analysis as well. Specifically, we first turn SAM into a high-quality pixel-level text segmentation (TS) model through a parameter-efficient fine-tuning approach. We use this TS model to iteratively generate the pixel-level text labels in a semi-automatical manner, unifying labels across the four text hierarchies in the HierText dataset. Subsequently, with these complete labels, we launch the end-to-end trainable Hi-SAM based on the TS architecture with a customized hierarchical mask decoder. During inference, Hi-SAM offers both automatic mask generation (AMG) mode and promptable segmentation (PS) mode. In the AMG mode, Hi-SAM segments pixel-level text foreground masks initially, then samples foreground points for hierarchical text mask generation and achieves layout analysis in passing. As for the PS mode, Hi-SAM provides word, text-line, and paragraph masks with a single point click. Experimental results show the state-of-the-art performance of our TS model: 84.86% fgIOU on Total-Text and 88.96% fgIOU on TextSeg for pixel-level text segmentation. Moreover, compared to the previous specialist for joint hierarchical detection and layout analysis on HierText, Hi-SAM achieves significant improvements: 4.73% PQ and 5.39% F1 on the text-line level, 5.49% PQ and 7.39% F1 on the paragraph level layout analysis, requiring $20\times$ fewer training epochs. The code is available at https://github.com/ymy-k/Hi-SAM.

cs.CV↗

arXiv2024

DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models

Recent text-to-image personalization methods have shown great promise in teaching a diffusion model user-specified concepts given a few images for reusing the acquired concepts in a novel context. With massive efforts being dedicated to personalized generation, a promising extension is personalized editing, namely to edit an image using personalized concepts, which can provide a more precise guidance signal than traditional textual guidance. To address this, a straightforward solution is to incorporate a personalized diffusion model with a text-driven editing framework. However, such a solution often shows unsatisfactory editability on the source image. To address this, we propose DreamSteerer, a plug-in method for augmenting existing T2I personalization methods. Specifically, we enhance the source image conditioned editability of a personalized diffusion model via a novel Editability Driven Score Distillation (EDSD) objective. Moreover, we identify a mode trapping issue with EDSD, and propose a mode shifting regularization with spatial feature guided sampling to avoid such an issue. We further employ two key modifications to the Delta Denoising Score framework that enable high-fidelity local editing with personalized concepts. Extensive experiments validate that DreamSteerer can significantly improve the editability of several T2I personalization baselines while being computationally efficient.

cs.CV↗