arXiv subjects

Chengzhi Liu

Publications and source records attributed to Chengzhi Liu.

32 records · Page 2Linked to original sources

arXiv2025

Multimodal Situational Safety

Multimodal Large Language Models (MLLMs) are rapidly evolving, demonstrating impressive capabilities as multimodal assistants that interact with both humans and their environments. However, this increased sophistication introduces significant safety concerns. In this paper, we present the first evaluation and analysis of a novel safety challenge termed Multimodal Situational Safety, which explores how safety considerations vary based on the specific situation in which the user or agent is engaged. We argue that for an MLLM to respond safely, whether through language or action, it often needs to assess the safety implications of a language query within its corresponding visual context. To evaluate this capability, we develop the Multimodal Situational Safety benchmark (MSSBench) to assess the situational safety performance of current MLLMs. The dataset comprises 1,820 language query-image pairs, half of which the image context is safe, and the other half is unsafe. We also develop an evaluation framework that analyzes key safety aspects, including explicit safety reasoning, visual understanding, and, crucially, situational safety reasoning. Our findings reveal that current MLLMs struggle with this nuanced safety problem in the instruction-following setting and struggle to tackle these situational safety challenges all at once, highlighting a key area for future research. Furthermore, we develop multi-agent pipelines to coordinately solve safety challenges, which shows consistent improvement in safety over the original MLLM response. Code and data: mssbench.github.io.

cs.AI↗

arXiv2025

PG-SAM: Prior-Guided SAM with Medical for Multi-organ Segmentation

Segment Anything Model (SAM) demonstrates powerful zero-shot capabilities; however, its accuracy and robustness significantly decrease when applied to medical image segmentation. Existing methods address this issue through modality fusion, integrating textual and image information to provide more detailed priors. In this study, we argue that the granularity of text and the domain gap affect the accuracy of the priors. Furthermore, the discrepancy between high-level abstract semantics and pixel-level boundary details in images can introduce noise into the fusion process. To address this, we propose Prior-Guided SAM (PG-SAM), which employs a fine-grained modality prior aligner to leverage specialized medical knowledge for better modality alignment. The core of our method lies in efficiently addressing the domain gap with fine-grained text from a medical LLM. Meanwhile, it also enhances the priors' quality after modality alignment, ensuring more accurate segmentation. In addition, our decoder enhances the model's expressive capabilities through multi-level feature fusion and iterative mask optimizer operations, supporting unprompted learning. We also propose a unified pipeline that effectively supplies high-quality semantic information to SAM. Extensive experiments on the Synapse dataset demonstrate that the proposed PG-SAM achieves state-of-the-art performance. Our code is released at https://github.com/logan-0623/PG-SAM.

cs.CV↗

arXiv2025

MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation

Recent multimodal large language models (MLLMs) have demonstrated significant potential in open-ended conversation, generating more accurate and personalized responses. However, their abilities to memorize, recall, and reason in sustained interactions within real-world scenarios remain underexplored. This paper introduces MMRC, a Multi-Modal Real-world Conversation benchmark for evaluating six core open-ended abilities of MLLMs: information extraction, multi-turn reasoning, information update, image management, memory recall, and answer refusal. With data collected from real-world scenarios, MMRC comprises 5,120 conversations and 28,720 corresponding manually labeled questions, posing a significant challenge to existing MLLMs. Evaluations on 20 MLLMs in MMRC indicate an accuracy drop during open-ended interactions. We identify four common failure patterns: long-term memory degradation, inadequacies in updating factual knowledge, accumulated assumption of error propagation, and reluctance to say no. To mitigate these issues, we propose a simple yet effective NOTE-TAKING strategy, which can record key information from the conversation and remind the model during its responses, enhancing conversational capabilities. Experiments across six MLLMs demonstrate significant performance improvements.

cs.CL↗

arXiv2025

Incomplete Modality Disentangled Representation for Ophthalmic Disease Grading and Diagnosis

Ophthalmologists typically require multimodal data sources to improve diagnostic accuracy in clinical decisions. However, due to medical device shortages, low-quality data and data privacy concerns, missing data modalities are common in real-world scenarios. Existing deep learning methods tend to address it by learning an implicit latent subspace representation for different modality combinations. We identify two significant limitations of these methods: (1) implicit representation constraints that hinder the model's ability to capture modality-specific information and (2) modality heterogeneity, causing distribution gaps and redundancy in feature representations. To address these, we propose an Incomplete Modality Disentangled Representation (IMDR) strategy, which disentangles features into explicit independent modal-common and modal-specific features by guidance of mutual information, distilling informative knowledge and enabling it to reconstruct valuable missing semantics and produce robust multimodal representations. Furthermore, we introduce a joint proxy learning module that assists IMDR in eliminating intra-modality redundancy by exploiting the extracted proxies from each class. Experiments on four ophthalmology multimodal datasets demonstrate that the proposed IMDR outperforms the state-of-the-art methods significantly.

cs.CV↗

arXiv2025

EMK-KEN: A High-Performance Approach for Assessing Knowledge Value in Citation Network

With the explosive growth of academic literature, effectively evaluating the knowledge value of literature has become quite essential. However, most of the existing methods focus on modeling the entire citation network, which is structurally complex and often suffers from long sequence dependencies when dealing with text embeddings. Thus, they might have low efficiency and poor robustness in different fields. To address these issues, a novel knowledge evaluation method is proposed, called EMK-KEN. The model consists of two modules. Specifically, the first module utilizes MetaFP and Mamba to capture semantic features of node metadata and text embeddings to learn contextual representations of each paper. The second module utilizes KAN to further capture the structural information of citation networks in order to learn the differences in different fields of networks. Extensive experiments based on ten benchmark datasets show that our method outperforms the state-of-the-art competitors in effectiveness and robustness.

cs.IR↗

arXiv2025

LAMOST medium-resolution spectroscopic survey of Galactic Open Clusters (LAMOST-MRS-O): An overview of survey plan and preliminary results

As part of the LAMOST medium-resolution spectroscopic survey, the LAMOST-MRS-O is a non-time domain survey that aims to perform medium-resolution spectral observations for member stars in the open cluster area. This survey plans to obtain the spectroscopic parameters such as radial velocity and metal abundances of member stars and provide data support for further study on the chemical and dynamical characteristics and evolution of open clusters in combination with Gaia data. We have completed the observations on ten open cluster fields and obtained 235184 medium-resolution spectra of 133792 stars. Based on the data analyzed of LAMOST DR11V1.1, for some clusters of particular concern, it is found that the sampling ratio of members stars with Gmag < 15 mag can reach 70%, which indicates that the LAMOST-MRS-O has reached our initial design goal.

astro-ph.GA↗

arXiv2024

Goal-guided Generative Prompt Injection Attack on Large Language Models

Current large language models (LLMs) provide a strong foundation for large-scale user-oriented natural language tasks. A large number of users can easily inject adversarial text or instructions through the user interface, thus causing LLMs model security challenges. Although there is currently a large amount of research on prompt injection attacks, most of these black-box attacks use heuristic strategies. It is unclear how these heuristic strategies relate to the success rate of attacks and thus effectively improve model robustness. To solve this problem, we redefine the goal of the attack: to maximize the KL divergence between the conditional probabilities of the clean text and the adversarial text. Furthermore, we prove that maximizing the KL divergence is equivalent to maximizing the Mahalanobis distance between the embedded representation $x$ and $x'$ of the clean text and the adversarial text when the conditional probability is a Gaussian distribution and gives a quantitative relationship on $x$ and $x'$. Then we designed a simple and effective goal-guided generative prompt injection strategy (G2PIA) to find an injection text that satisfies specific constraints to achieve the optimal attack effect approximately. It is particularly noteworthy that our attack method is a query-free black-box attack method with low computational cost. Experimental results on seven LLM models and four datasets show the effectiveness of our attack method.

cs.CR↗

arXiv2024

Multi-task Prompt Words Learning for Social Media Content Generation

The rapid development of the Internet has profoundly changed human life. Humans are increasingly expressing themselves and interacting with others on social media platforms. However, although artificial intelligence technology has been widely used in many aspects of life, its application in social media content creation is still blank. To solve this problem, we propose a new prompt word generation framework based on multi-modal information fusion, which combines multiple tasks including topic classification, sentiment analysis, scene recognition and keyword extraction to generate more comprehensive prompt words. Subsequently, we use a template containing a set of prompt words to guide ChatGPT to generate high-quality tweets. Furthermore, in the absence of effective and objective evaluation criteria in the field of content generation, we use the ChatGPT tool to evaluate the results generated by the algorithm, making large-scale evaluation of content generation algorithms possible. Evaluation results on extensive content generation demonstrate that our cue word generation framework generates higher quality content compared to manual methods and other cueing techniques, while topic classification, sentiment analysis, and scene recognition significantly enhance content clarity and its consistency with the image.

cs.CL↗

arXiv2024

Better Sampling, towards Better End-to-end Small Object Detection

While deep learning-based general object detection has made significant strides in recent years, the effectiveness and efficiency of small object detection remain unsatisfactory. This is primarily attributed not only to the limited characteristics of such small targets but also to the high density and mutual overlap among these targets. The existing transformer-based small object detectors do not leverage the gap between accuracy and inference speed. To address challenges, we propose methods enhancing sampling within an end-to-end framework. Sample Points Refinement (SPR) constrains localization and attention, preserving meaningful interactions in the region of interest and filtering out misleading information. Scale-aligned Target (ST) integrates scale information into target confidence, improving classification for small object detection. A task-decoupled Sample Reweighting (SR) mechanism guides attention toward challenging positive examples, utilizing a weight generator module to assess the difficulty and adjust classification loss based on decoder layer outcomes. Comprehensive experiments across various benchmarks reveal that our proposed detector excels in detecting small objects. Our model demonstrates a significant enhancement, achieving a 2.9\% increase in average precision (AP) over the state-of-the-art (SOTA) on the VisDrone dataset and a 1.7\% improvement on the SODA-D dataset.

cs.CV↗

arXiv2024

MC-DBN: A Deep Belief Network-Based Model for Modality Completion

Recent advancements in multi-modal artificial intelligence (AI) have revolutionized the fields of stock market forecasting and heart rate monitoring. Utilizing diverse data sources can substantially improve prediction accuracy. Nonetheless, additional data may not always align with the original dataset. Interpolation methods are commonly utilized for handling missing values in modal data, though they may exhibit limitations in the context of sparse information. Addressing this challenge, we propose a Modality Completion Deep Belief Network-Based Model (MC-DBN). This approach utilizes implicit features of complete data to compensate for gaps between itself and additional incomplete data. It ensures that the enhanced multi-modal data closely aligns with the dynamic nature of the real world to enhance the effectiveness of the model. We conduct evaluations of the MC-DBN model in two datasets from the stock market forecasting and heart rate monitoring domains. Comprehensive experiments showcase the model's capacity to bridge the semantic divide present in multi-modal data, subsequently enhancing its performance. The source code is available at: https://github.com/logan-0623/DBN-generate

cs.LG↗

arXiv2024

MTSA-SNN: A Multi-modal Time Series Analysis Model Based on Spiking Neural Network

Time series analysis and modelling constitute a crucial research area. Traditional artificial neural networks struggle with complex, non-stationary time series data due to high computational complexity, limited ability to capture temporal information, and difficulty in handling event-driven data. To address these challenges, we propose a Multi-modal Time Series Analysis Model Based on Spiking Neural Network (MTSA-SNN). The Pulse Encoder unifies the encoding of temporal images and sequential information in a common pulse-based representation. The Joint Learning Module employs a joint learning function and weight allocation mechanism to fuse information from multi-modal pulse signals complementary. Additionally, we incorporate wavelet transform operations to enhance the model's ability to analyze and evaluate temporal information. Experimental results demonstrate that our method achieved superior performance on three complex time-series tasks. This work provides an effective event-driven approach to overcome the challenges associated with analyzing intricate temporal information. Access to the source code is available at https://github.com/Chenngzz/MTSA-SNN}{https://github.com/Chenngzz/MTSA-SNN

cs.CV↗

arXiv2023

Preconditioned geometric iterative methods for cubic B-spline interpolation curves

The geometric iterative method (GIM) is widely used in data interpolation/fitting, but its slow convergence affects the computational efficiency. Recently, much work was done to guarantee the acceleration of GIM in the literature. In this work, we aim to further accelerate the rate of convergence by introducing a preconditioning technique. After constructing the preconditioner, we preprocess the progressive iterative approximation (PIA) and its variants, called the preconditioned GIMs. We show that the proposed preconditioned GIMs converge and the extra computation cost brought by the preconditioning technique is negligible. Several numerical experiments are given to demonstrate that our preconditioner can accelerate the convergence rate of PIA and its variants.

math.NA↗

arXiv2022

Randomized progressive iterative approximation for B-spline curve and surface fittings

For large-scale data fitting, the least-squares progressive iterative approximation is a widely used method in many applied domains because of its intuitive geometric meaning and efficiency. In this work, we present a randomized progressive iterative approximation (RPIA) for the B-spline curve and surface fittings. In each iteration, RPIA locally adjusts the control points according to a random criterion of index selections. The difference for each control point is computed concerning the randomized block coordinate descent method. From geometric and algebraic aspects, the illustrations of RPIA are provided. We prove that RPIA constructs a series of fitting curves (resp., surfaces), whose limit curve (resp., surface) can converge in expectation to the least-squares fitting result of the given data points. Numerical experiments are given to confirm our results and show the benefits of RPIA.

math.NA↗

arXiv2022

On the extended randomized multiple row method for solving linear least-squares problems

The randomized row method is a popular representative of the iterative algorithm because of its efficiency in solving the overdetermined and consistent systems of linear equations. In this paper, we present an extended randomized multiple row method to solve a given overdetermined and inconsistent linear system and analyze its computational complexities at each iteration. We prove that the proposed method can linearly converge in the mean square to the least-squares solution with a minimum Euclidean norm. Several numerical studies are presented to corroborate our theoretical findings. The real-world applications, such as image reconstruction and large noisy data fitting in computer-aided geometric design, are also presented for illustration purposes.

math.NA↗