arXiv subjects

Jun Liu

Publications and source records attributed to Jun Liu.

At least 361 records · Page 20Linked to original sources

arXiv2025

MyESL: Sparse learning in molecular evolution and phylogenetic analysis

Evolutionary sparse learning (ESL) uses a supervised machine learning approach, Least Absolute Shrinkage and Selection Operator (LASSO), to build models explaining the relationship between a hypothesis and the variation across genomic features (e.g., sites) in sequence alignments. ESL employs sparsity between and within the groups of genomic features (e.g., genomic loci) by using sparse-group LASSO. Although some software packages are available for performing sparse group LASSO, we found them less well-suited for processing and analyzing genome-scale data containing millions of features, such as bases. MyESL software fills the need for open-source software for conducting ESL analyses with facilities to pre-process the input hypotheses and large alignments, make LASSO flexible and computationally efficient, and post-process the output model to produce different metrics useful in functional or evolutionary genomics. MyESL can take phylogenetic trees and sequence alignments as input and transform them into numeric responses and features, respecetively. The model outputs are processed into user-friendly text and graphical files. The computational core of MyESL is written in C++, which offers model building with or without group sparsity, while the pre- and post-processing of inputs and model outputs is performed using customized functions written in Python. One of its applications in phylogenomics showcases the utility of MyESL. Our analysis of empirical genome-scale datasets shows that MyESL can build evolutionary models quickly and efficiently on a personal desktop, while other computational packages were unable due to their prohibitive requirements of computational resources and time. MyESL is available for Python environments on Linux and distributed as a standalone application for Windows and macOS. It is available from https://github.com/kumarlabgit/MyESL.

q-bio.PE↗

arXiv2025

Inconsistencies in Unstructured Geometric Volume-of-Fluid Methods for Two-Phase Flows with High Density Ratios

Geometric flux-based Volume-of-Fluid (VOF) methods are widely considered consistent in handling two-phase flows with high density ratios. However, although the conservation of mass and momentum is consistent for two-phase incompressible single-field Navier-Stokes equations without phase-change, discretization may easily introduce small inconsistencies that result in very large errors or catastrophic failure. We apply the consistency conditions derived for the unstructured Level Set / Front Tracking method to flux-based geometric VOF methods, and implement our discretization into the plicRDF-isoAdvector geometrical VOF method. We find that computing the mass flux by scaling the geometrically computed fluxed phase-specific volume destroys the equivalence between the scaled volume fraction equation and the mass conservation equation, depending on the choice for the temporal and convective term discretization schemes. We propose two solutions. First, based on the analysis of discretization errors, we suggest a consistent combination of the temporal discretization scheme and the interpolation scheme for the momentum convection term. Second, similar to our previous work on the unstructured Level Set / Front Tracking method, we solve an auxiliary mass conservation equation with a geometrical calculation of the face-centered density. We prove the equivalence between these two approaches mathematically and verify and validate their numerical stability for density ratios in the range $[1,10^6]$ and viscosity ratios in the range $[10^2,10^5]$.

physics.comp-ph↗

arXiv2025

Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension

In this work, we address the challenging task of Generalized Referring Expression Comprehension (GREC). Compared to the classic Referring Expression Comprehension (REC) that focuses on single-target expressions, GREC extends the scope to a more practical setting by further encompassing no-target and multi-target expressions. Existing REC methods face challenges in handling the complex cases encountered in GREC, primarily due to their fixed output and limitations in multi-modal representations. To address these issues, we propose a Hierarchical Alignment-enhanced Adaptive Grounding Network (HieA2G) for GREC, which can flexibly deal with various types of referring expressions. First, a Hierarchical Multi-modal Semantic Alignment (HMSA) module is proposed to incorporate three levels of alignments, including word-object, phrase-object, and text-image alignment. It enables hierarchical cross-modal interactions across multiple levels to achieve comprehensive and robust multi-modal understanding, greatly enhancing grounding ability for complex cases. Then, to address the varying number of target objects in GREC, we introduce an Adaptive Grounding Counter (AGC) to dynamically determine the number of output targets. Additionally, an auxiliary contrastive loss is employed in AGC to enhance object-counting ability by pulling in multi-modal features with the same counting and pushing away those with different counting. Extensive experimental results show that HieA2G achieves new state-of-the-art performance on the challenging GREC task and also the other 4 tasks, including REC, Phrase Grounding, Referring Expression Segmentation (RES), and Generalized Referring Expression Segmentation (GRES), demonstrating the remarkable superiority and generalizability of the proposed HieA2G.

cs.CV↗

arXiv2024

Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices

Federated fine-tuning (FedFT) has been proposed to fine-tune the pre-trained language models in a distributed manner. However, there are two critical challenges for efficient FedFT in practical applications, i.e., resource constraints and system heterogeneity. Existing works rely on parameter-efficient fine-tuning methods, e.g., low-rank adaptation (LoRA), but with major limitations. Herein, based on the inherent characteristics of FedFT, we observe that LoRA layers with higher ranks added close to the output help to save resource consumption while achieving comparable fine-tuning performance. Then we propose a novel LoRA-based FedFT framework, termed LEGEND, which faces the difficulty of determining the number of LoRA layers (called, LoRA depth) and the rank of each LoRA layer (called, rank distribution). We analyze the coupled relationship between LoRA depth and rank distribution, and design an efficient LoRA configuration algorithm for heterogeneous devices, thereby promoting fine-tuning efficiency. Extensive experiments are conducted on a physical platform with 80 commercial devices. The results show that LEGEND can achieve a speedup of 1.5-2.8$\times$ and save communication costs by about 42.3% when achieving the target accuracy, compared to the advanced solutions.

cs.DC↗

arXiv2024

Baichuan-Omni Technical Report

The salient multimodal capabilities and interactive experience of GPT-4o highlight its critical role in practical applications, yet it lacks a high-performing open-source counterpart. In this paper, we introduce Baichuan-omni, the first open-source 7B Multimodal Large Language Model (MLLM) adept at concurrently processing and analyzing modalities of image, video, audio, and text, while delivering an advanced multimodal interactive experience and strong performance. We propose an effective multimodal training schema starting with 7B model and proceeding through two stages of multimodal alignment and multitask fine-tuning across audio, image, video, and text modal. This approach equips the language model with the ability to handle visual and audio data effectively. Demonstrating strong performance across various omni-modal and multimodal benchmarks, we aim for this contribution to serve as a competitive baseline for the open-source community in advancing multimodal understanding and real-time interaction.

cs.AI↗

arXiv2024

STARVERI: Efficient and Accurate Verification for Risk-Avoidance Routing in LEO Satellite Networks

Emerging satellite Internet constellations such as SpaceX's Starlink will deploy thousands of broadband satellites and construct Low-Earth Orbit(LEO) satellite networks(LSNs) in space, significantly expanding the boundaries of today's terrestrial Internet. However, due to the unique global LEO dynamics, satellite routers will inevitably pass through uncontrolled areas, suffering from security threats. It should be important for satellite network operators(SNOs) to enable verifiable risk-avoidance routing to identify path anomalies. In this paper, we present STARVERI, a novel network path verification framework tailored for emerging LSNs. STARVERI addresses the limitations of existing crypto-based and delay-based verification approaches and accomplishes efficient and accurate path verification by: (i) adopting a dynamic relay selection mechanism deployed in SNO's operation center to judiciously select verifiable relays for each communication pair over LSNs; and (ii) incorporating a lightweight path verification algorithm to dynamically verify each segment path split by distributed relays. We build an LSN simulator based on real constellation information and the results demonstrate that STARVERI can significantly improve the path verification accuracy and achieve lower router overhead compared with existing approaches.

cs.NI↗

arXiv2024

Mask-RadarNet: Enhancing Transformer With Spatial-Temporal Semantic Context for Radar Object Detection in Autonomous Driving

As a cost-effective and robust technology, automotive radar has seen steady improvement during the last years, making it an appealing complement to commonly used sensors like camera and LiDAR in autonomous driving. Radio frequency data with rich semantic information are attracting more and more attention. Most current radar-based models take radio frequency image sequences as the input. However, these models heavily rely on convolutional neural networks and leave out the spatial-temporal semantic context during the encoding stage. To solve these problems, we propose a model called Mask-RadarNet to fully utilize the hierarchical semantic features from the input radar data. Mask-RadarNet exploits the combination of interleaved convolution and attention operations to replace the traditional architecture in transformer-based models. In addition, patch shift is introduced to the Mask-RadarNet for efficient spatial-temporal feature learning. By shifting part of patches with a specific mosaic pattern in the temporal dimension, Mask-RadarNet achieves competitive performance while reducing the computational burden of the spatial-temporal modeling. In order to capture the spatial-temporal semantic contextual information, we design the class masking attention module (CMAM) in our encoder. Moreover, a lightweight auxiliary decoder is added to our model to aggregate prior maps generated from the CMAM. Experiments on the CRUW dataset demonstrate the superiority of the proposed method to some state-of-the-art radar-based object detection algorithms. With relatively lower computational complexity and fewer parameters, the proposed Mask-RadarNet achieves higher recognition accuracy for object detection in autonomous driving.

cs.CV↗

arXiv2024

Urban Air Temperature Prediction using Conditional Diffusion Models

Urbanization as a global trend has led to many environmental challenges, including the urban heat island (UHI) effect. The increase in temperature has a significant impact on the well-being of urban residents. Air temperature ($T_a$) at 2m above the surface is a key indicator of the UHI effect. How land use land cover (LULC) affects $T_a$ is a critical research question which requires high-resolution (HR) $T_a$ data at neighborhood scale. However, weather stations providing $T_a$ measurements are sparsely distributed e.g. more than 10km apart; and numerical models are impractically slow and computationally expensive. In this work, we propose a novel method to predict HR $T_a$ at 100m ground separation distance (gsd) using land surface temperature (LST) and other LULC related features which can be easily obtained from satellite imagery. Our method leverages diffusion models for the first time to generate accurate and visually realistic HR $T_a$ maps, which outperforms prior methods. We pave the way for meteorological research using computer vision techniques by providing a dataset of an extended spatial and temporal coverage, and a high spatial resolution as a benchmark for future research. Furthermore, we show that our model can be applied to urban planning by simulating the impact of different urban designs on $T_a$.

cs.CV↗

arXiv2024

A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future

Artificial intelligence (AI) has rapidly developed through advancements in computational power and the growth of massive datasets. However, this progress has also heightened challenges in interpreting the "black-box" nature of AI models. To address these concerns, eXplainable AI (XAI) has emerged with a focus on transparency and interpretability to enhance human understanding and trust in AI decision-making processes. In the context of multimodal data fusion and complex reasoning scenarios, the proposal of Multimodal eXplainable AI (MXAI) integrates multiple modalities for prediction and explanation tasks. Meanwhile, the advent of Large Language Models (LLMs) has led to remarkable breakthroughs in natural language processing, yet their complexity has further exacerbated the issue of MXAI. To gain key insights into the development of MXAI methods and provide crucial guidance for building more transparent, fair, and trustworthy AI systems, we review the MXAI methods from a historical perspective and categorize them across four eras: traditional machine learning, deep learning, discriminative foundation models, and generative LLMs. We also review evaluation metrics and datasets used in MXAI research, concluding with a discussion of future challenges and directions. A project related to this review has been created at https://github.com/ShilinSun/mxai_review.

cs.CV↗

arXiv2024

Brain-inspired AI Agent: The Way Towards AGI

Artificial General Intelligence (AGI), widely regarded as the fundamental goal of artificial intelligence, represents the realization of cognitive capabilities that enable the handling of general tasks with human-like proficiency. Researchers in brain-inspired AI seek inspiration from the operational mechanisms of the human brain, aiming to replicate its functional rules in intelligent models. Moreover, with the rapid development of large-scale models in recent years, the concept of agents has garnered increasing attention, with researchers widely recognizing it as a necessary pathway toward achieving AGI. In this article, we propose the concept of a brain-inspired AI agent and analyze how to extract relatively feasible and agent-compatible cortical region functionalities and their associated functional connectivity networks from the complex mechanisms of the human brain. Implementing these structures within an agent enables it to achieve basic cognitive intelligence akin to human capabilities. Finally, we explore the limitations and challenges for realizing brain-inspired agents and discuss their future development.

cs.NE↗

arXiv2024

Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomaly

Recent advancements in video anomaly understanding (VAU) have opened the door to groundbreaking applications in various fields, such as traffic monitoring and industrial automation. While the current benchmarks in VAU predominantly emphasize the detection and localization of anomalies. Here, we endeavor to delve deeper into the practical aspects of VAU by addressing the essential questions: "what anomaly occurred?", "why did it happen?", and "how severe is this abnormal event?". In pursuit of these answers, we introduce a comprehensive benchmark for Exploring the Causation of Video Anomalies (ECVA). Our benchmark is meticulously designed, with each video accompanied by detailed human annotations. Specifically, each instance of our ECVA involves three sets of human annotations to indicate "what", "why" and "how" of an anomaly, including 1) anomaly type, start and end times, and event descriptions, 2) natural language explanations for the cause of an anomaly, and 3) free text reflecting the effect of the abnormality. Building upon this foundation, we propose a novel prompt-based methodology that serves as a baseline for tackling the intricate challenges posed by ECVA. We utilize "hard prompt" to guide the model to focus on the critical parts related to video anomaly segments, and "soft prompt" to establish temporal and spatial relationships within these anomaly segments. Furthermore, we propose AnomEval, a specialized evaluation metric crafted to align closely with human judgment criteria for ECVA. This metric leverages the unique features of the ECVA dataset to provide a more comprehensive and reliable assessment of various video large language models. We demonstrate the efficacy of our approach through rigorous experimental analysis and delineate possible avenues for further investigation into the comprehension of video anomaly causation.

cs.CV↗

arXiv2024

Reduction in Thermal Conductivity of Monolayer MoS2 by Large Mechanical Strains for Efficient Thermal Management

Two dimensional (2D) materials such as graphene and transition metal dichalcogenides (TMDC) have received extensive research interests and investigations in the past decade. In this research, we report the first experimental measurement of the in plane thermal conductivity of MoS2 monolayer under a large mechanical strain using optothermal Raman technique. This measurement technique is direct without additional processing to the material, and MoS2's absorption coefficient is discovered during the measurement process to further increase this technique's precision. Tunable uniaxial tensile strains are applied on the MoS2 monolayer by stretching a flexible substrate it sits on. Experimental results demonstrate that, the thermal conductivity is substantially suppressed by tensile strains: under the tensile strain of 6.3%, the thermal conductivity of the MoS2 monolayer drops approximately by 62%. A serious of thermal transport properties at a group of mechanical strains are also reported, presenting a strain dependent trend. It is the first and original study of 2D materials' thermal transport properties under a large mechanical strain, and provides important information that the thermal transport of MoS2 will significantly decrease at a large mechanical strain. This finding provides the key information for flexible and wearable electronics thermal management and designs.

cond-mat.mtrl-sci↗

arXiv2024

Data-driven optimal control of unknown nonlinear dynamical systems using the Koopman operator

Nonlinear optimal control is vital for numerous applications but remains challenging for unknown systems due to the difficulties in accurately modelling dynamics and handling computational demands, particularly in high-dimensional settings. This work develops a theoretically certifiable framework that integrates a modified Koopman operator approach with model-based reinforcement learning to address these challenges. By relaxing the requirements on observable functions, our method incorporates nonlinear terms involving both states and control inputs, significantly enhancing system identification accuracy. Moreover, by leveraging the power of neural networks to solve partial differential equations (PDEs), our approach is able to achieving stabilizing control for high-dimensional dynamical systems, up to 9-dimensional. The learned value function and control laws are proven to converge to those of the true system at each iteration. Additionally, the accumulated cost of the learned control closely approximates that of the true system, with errors ranging from $10^{-5}$ to $10^{-3}$.

eess.SY↗

arXiv2024

Precise Facial Landmark Detection by Dynamic Semantic Aggregation Transformer

At present, deep neural network methods have played a dominant role in face alignment field. However, they generally use predefined network structures to predict landmarks, which tends to learn general features and leads to mediocre performance, e.g., they perform well on neutral samples but struggle with faces exhibiting large poses or occlusions. Moreover, they cannot effectively deal with semantic gaps and ambiguities among features at different scales, which may hinder them from learning efficient features. To address the above issues, in this paper, we propose a Dynamic Semantic-Aggregation Transformer (DSAT) for more discriminative and representative feature (i.e., specialized feature) learning. Specifically, a Dynamic Semantic-Aware (DSA) model is first proposed to partition samples into subsets and activate the specific pathways for them by estimating the semantic correlations of feature channels, making it possible to learn specialized features from each subset. Then, a novel Dynamic Semantic Specialization (DSS) model is designed to mine the homogeneous information from features at different scales for eliminating the semantic gap and ambiguities and enhancing the representation ability. Finally, by integrating the DSA model and DSS model into our proposed DSAT in both dynamic architecture and dynamic parameter manners, more specialized features can be learned for achieving more precise face alignment. It is interesting to show that harder samples can be handled by activating more feature channels. Extensive experiments on popular face alignment datasets demonstrate that our proposed DSAT outperforms state-of-the-art models in the literature.Our code is available at https://github.com/GERMINO-LiuHe/DSAT.

cs.CV↗

arXiv2024

Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models

Despite inheriting security measures from underlying language models, Vision-Language Models (VLMs) may still be vulnerable to safety alignment issues. Through empirical analysis, we uncover two critical findings: scenario-matched images can significantly amplify harmful outputs, and contrary to common assumptions in gradient-based attacks, minimal loss values do not guarantee optimal attack effectiveness. Building on these insights, we introduce MLAI (Multi-Loss Adversarial Images), a novel jailbreak framework that leverages scenario-aware image generation for semantic alignment, exploits flat minima theory for robust adversarial image selection, and employs multi-image collaborative attacks for enhanced effectiveness. Extensive experiments demonstrate MLAI's significant impact, achieving attack success rates of 77.75% on MiniGPT-4 and 82.80% on LLaVA-2, substantially outperforming existing methods by margins of 34.37% and 12.77% respectively. Furthermore, MLAI shows considerable transferability to commercial black-box VLMs, achieving up to 60.11% success rate. Our work reveals fundamental visual vulnerabilities in current VLMs safety mechanisms and underscores the need for stronger defenses. Warning: This paper contains potentially harmful example text.

cs.CV↗

arXiv2024

Polarization and Orbital Angular Momentum Encoded Quantum Toffoli Gate Enabled by Diffractive Neural Networks

Controlled quantum gates play a crucial role in enabling quantum universal operations by facilitating interactions between qubits. Direct implementation of three-qubit gates simplifies the design of quantum circuits, thereby being conducive to performing complex quantum algorithms. Here, we propose and present an experimental demonstration of a quantum Toffoli gate fully exploiting the polarization and orbital angular momentum of a single photon. The Toffoli gate is implemented using the polarized diffractive neural networks scheme, achieving a mean truth table visibility of $97.27\pm0.20\%$. We characterize the gate's performance through quantum state tomography on 216 different input states and quantum process tomography, which yields a process fidelity of $94.05\pm 0.02\%$. Our method offers a novel approach for realizing the Toffoli gate without requiring exponential optical elements while maintaining extensibility to the implementation of other three-qubit gates.

quant-ph↗

arXiv2024

Ultrahigh-fidelity spatial mode quantum gates in high-dimensional space by diffractive deep neural networks

While the spatial mode of photons is widely used in quantum cryptography, its potential for quantum computation remains largely unexplored. Here, we showcase the use of the multi-dimensional spatial mode of photons to construct a series of high-dimensional quantum gates, achieved through the use of diffractive deep neural networks (D2NNs). Notably, our gates demonstrate high fidelity of up to 99.6(2)%, as characterized by quantum process tomography. Our experimental implementation of these gates involves a programmable array of phase layers in a compact and scalable device, capable of performing complex operations or even quantum circuits. We also demonstrate the efficacy of the D2NN gates by successfully implementing the Deutsch algorithm and propose an intelligent deployment protocol that involves self-configuration and self-optimization. Moreover, we conduct a comparative analysis of the D2NN gate's performance to the wave-front matching approach. Overall, our work opens a door for designing specific quantum gates using deep learning, with the potential for reliable execution of quantum computation.

quant-ph↗

arXiv2024

Artificial General Intelligence for Medical Imaging Analysis

Large-scale Artificial General Intelligence (AGI) models, including Large Language Models (LLMs) such as ChatGPT/GPT-4, have achieved unprecedented success in a variety of general domain tasks. Yet, when applied directly to specialized domains like medical imaging, which require in-depth expertise, these models face notable challenges arising from the medical field's inherent complexities and unique characteristics. In this review, we delve into the potential applications of AGI models in medical imaging and healthcare, with a primary focus on LLMs, Large Vision Models, and Large Multimodal Models. We provide a thorough overview of the key features and enabling techniques of LLMs and AGI, and further examine the roadmaps guiding the evolution and implementation of AGI models in the medical sector, summarizing their present applications, potentialities, and associated challenges. In addition, we highlight potential future research directions, offering a holistic view on upcoming ventures. This comprehensive review aims to offer insights into the future implications of AGI in medical imaging, healthcare, and beyond.

cs.AI↗