arXiv subjects

Jianing Li

Publications and source records attributed to Jianing Li.

At least 55 records · Page 3Linked to original sources

arXiv2025

Transport coefficients of dense nucleon matter at low temperature

The transport property of cold and dense nucleon matter is important for nuclear physics but is relatively less studied than that at finite temperatures. In this paper, we present a primary study of bulk and shear viscosities in the limit $T/μ_B \ll 1$, where $T$ and $μ_B$ are the temperature and the baryon chemical potential. The analysis is performed for a generic system where nucleons are dressed by the condensation of both scalar and vector interactions. Under the relaxation time approximation of the Boltzmann equation, we compute the viscosities of the system to leading power in $T/μ_B$ expansion and establish a relation between the thermodynamic potential and transport coefficients, including bulk viscosity ($ζ$) and shear viscosity ($η$). It is found that hydrodynamic stability ($ζ>0$) imposes additional constraints on the thermodynamic potential. As an example, these relations are applied to the Walecka model. The fluid properties of the cold and dense nucleon matter are characterized by the dimensionless combination of viscosities times the quasi-Fermi momentum over the enthalpy. Furthermore, we discuss the implication of the stability condition on the range of applicability of the model.

nucl-th↗

arXiv2025

Towards Understanding How Knowledge Evolves in Large Vision-Language Models

Large Vision-Language Models (LVLMs) are gradually becoming the foundation for many artificial intelligence applications. However, understanding their internal working mechanisms has continued to puzzle researchers, which in turn limits the further enhancement of their capabilities. In this paper, we seek to investigate how multimodal knowledge evolves and eventually induces natural languages in LVLMs. We design a series of novel strategies for analyzing internal knowledge within LVLMs, and delve into the evolution of multimodal knowledge from three levels, including single token probabilities, token probability distributions, and feature encodings. In this process, we identify two key nodes in knowledge evolution: the critical layers and the mutation layers, dividing the evolution process into three stages: rapid evolution, stabilization, and mutation. Our research is the first to reveal the trajectory of knowledge evolution in LVLMs, providing a fresh perspective for understanding their underlying mechanisms. Our codes are available at https://github.com/XIAO4579/Vlm-interpretability.

cs.CV↗

arXiv2025

SliceOcc: Indoor 3D Semantic Occupancy Prediction with Vertical Slice Representation

3D semantic occupancy prediction is a crucial task in visual perception, as it requires the simultaneous comprehension of both scene geometry and semantics. It plays a crucial role in understanding 3D scenes and has great potential for various applications, such as robotic vision perception and autonomous driving. Many existing works utilize planar-based representations such as Bird's Eye View (BEV) and Tri-Perspective View (TPV). These representations aim to simplify the complexity of 3D scenes while preserving essential object information, thereby facilitating efficient scene representation. However, in dense indoor environments with prevalent occlusions, directly applying these planar-based methods often leads to difficulties in capturing global semantic occupancy, ultimately degrading model performance. In this paper, we present a new vertical slice representation that divides the scene along the vertical axis and projects spatial point features onto the nearest pair of parallel planes. To utilize these slice features, we propose SliceOcc, an RGB camera-based model specifically tailored for indoor 3D semantic occupancy prediction. SliceOcc utilizes pairs of slice queries and cross-attention mechanisms to extract planar features from input images. These local planar features are then fused to form a global scene representation, which is employed for indoor occupancy prediction. Experimental results on the EmbodiedScan dataset demonstrate that SliceOcc achieves a mIoU of 15.45% across 81 indoor categories, setting a new state-of-the-art performance among RGB camera-based models for indoor 3D semantic occupancy prediction. Code is available at https://github.com/NorthSummer/SliceOcc.

cs.CV↗

arXiv2024

AFM Cantilever Magnetometry for Measuring Femto-Nm Torques Generated by Single Magnetic Particles for Cell Actuation

Particles with high anisotropy in their magnetic properties and shape are of increasing interest for mechanobiology, where transducing a remotely applied magnetic field vector to a local mechanical response is crucial. An outstanding challenge is quantifying the mechanical torque of a single nanoparticle, typically in the range of atto- to femto-Newton-meters (Nm). The magneto-mechanical torque manifests due to a misalignment of the external magnetic field vector with the built-in magnetic anisotropy axis, as opposed to a magnetic force, and complicates the measurement scheme. In this work, we developed a method using a commercially available Atomic Force Microscopy setup and cantilevers to quantify the torque generated by a single synthetic antiferromagnetic (SAF) nanoplatelet with high perpendicular magnetic anisotropy. Specifically, we measured 1.6$\pm$0.6$\cdot$10$^{-15}$ Nm torque while applying 373$\pm$5 mT field at 12$\pm$2 degrees to the built-in anisotropy axis exerted by a single circular SAF nanoplatelet with 1.88 $μ$m diameter and 72 nm thickness, naively translating to a $\approx$ 1.7 nN maximum force at the nanoplatelet apex. This measured torque and derived force of the SAF nanoplatelets is strong enough for most applications in mechanobiology; for example, it can be used to rupture (cancer) cell membranes. Moreover, SAF nanoplatelets open a route for easy tuning of the built-in magnetic anisotropy and size, reducing the torque and allowing for small mechanical stimuli for ion channel activation. This work presents a straightforward and widely applicable method for characterizing magnetic particles' mechanical transduction, which is applied to SAF nanoplatelets with a high PMA.

cond-mat.mes-hall↗

arXiv2024

EventGPT: Event Stream Understanding with Multimodal Large Language Models

Event cameras record visual information as asynchronous pixel change streams, excelling at scene perception under unsatisfactory lighting or high-dynamic conditions. Existing multimodal large language models (MLLMs) concentrate on natural RGB images, failing in scenarios where event data fits better. In this paper, we introduce EventGPT, the first MLLM for event stream understanding, to the best of our knowledge, marking a pioneering attempt to integrate large language models (LLMs) with event stream comprehension. To mitigate the huge domain gaps, we develop a three-stage optimization paradigm to gradually equip a pre-trained LLM with the capability of understanding event-based scenes. Our EventGPT comprises an event encoder, followed by a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Firstly, RGB image-text pairs generated by GPT are leveraged to warm up the linear projector, referring to LLaVA, as the gap between natural image and language modalities is relatively smaller. Secondly, we construct a synthetic yet large dataset, N-ImageNet-Chat, consisting of event frames and corresponding texts to enable the use of the spatio-temporal aggregator and to train the event-language adapter, thereby aligning event features more closely with the language space. Finally, we gather an instruction dataset, Event-Chat, which contains extensive real-world data to fine-tune the entire model, further enhancing its generalization ability. We construct a comprehensive benchmark, and experiments show that EventGPT surpasses previous state-of-the-art MLLMs in generation quality, descriptive accuracy, and reasoning capability.

cs.CV↗

arXiv2024

HDI-Former: Hybrid Dynamic Interaction ANN-SNN Transformer for Object Detection Using Frames and Events

Combining the complementary benefits of frames and events has been widely used for object detection in challenging scenarios. However, most object detection methods use two independent Artificial Neural Network (ANN) branches, limiting cross-modality information interaction across the two visual streams and encountering challenges in extracting temporal cues from event streams with low power consumption. To address these challenges, we propose HDI-Former, a Hybrid Dynamic Interaction ANN-SNN Transformer, marking the first trial to design a directly trained hybrid ANN-SNN architecture for high-accuracy and energy-efficient object detection using frames and events. Technically, we first present a novel semantic-enhanced self-attention mechanism that strengthens the correlation between image encoding tokens within the ANN Transformer branch for better performance. Then, we design a Spiking Swin Transformer branch to model temporal cues from event streams with low power consumption. Finally, we propose a bio-inspired dynamic interaction mechanism between ANN and SNN sub-networks for cross-modality information interaction. The results demonstrate that our HDI-Former outperforms eleven state-of-the-art methods and our four baselines by a large margin. Our SNN branch also shows comparable performance to the ANN with the same architecture while consuming 10.57$\times$ less energy on the DSEC-Detection dataset. Our open-source code is available in the supplementary material.

cs.CV↗

arXiv2024

Classifier Guidance Enhances Diffusion-based Adversarial Purification by Preserving Predictive Information

Adversarial purification is one of the promising approaches to defend neural networks against adversarial attacks. Recently, methods utilizing diffusion probabilistic models have achieved great success for adversarial purification in image classification tasks. However, such methods fall into the dilemma of balancing the needs for noise removal and information preservation. This paper points out that existing adversarial purification methods based on diffusion models gradually lose sample information during the core denoising process, causing occasional label shift in subsequent classification tasks. As a remedy, we suggest to suppress such information loss by introducing guidance from the classifier confidence. Specifically, we propose Classifier-cOnfidence gUided Purification (COUP) algorithm, which purifies adversarial examples while keeping away from the classifier decision boundary. Experimental results show that COUP can achieve better adversarial robustness under strong attack methods.

cs.CV↗

arXiv2024

3D-UGCN: A Unified Graph Convolutional Network for Robust 3D Human Pose Estimation from Monocular RGB Images

Human pose estimation remains a multifaceted challenge in computer vision, pivotal across diverse domains such as behavior recognition, human-computer interaction, and pedestrian tracking. This paper proposes an improved method based on the spatial-temporal graph convolution net-work (UGCN) to address the issue of missing human posture skeleton sequences in single-view videos. We present the improved UGCN, which allows the network to process 3D human pose data and improves the 3D human pose skeleton sequence, thereby resolving the occlusion issue.

cs.CV↗

arXiv2024

Dissecting Multiplication in Transformers: Insights into LLMs

Transformer-based large language models have achieved remarkable performance across various natural language processing tasks. However, they often struggle with seemingly easy tasks like arithmetic despite their vast capabilities. This stark disparity raise human's concerns about their safe and ethical use, hinder their widespread adoption.In this paper, we focus on a typical arithmetic task, integer multiplication, to explore and explain the imperfection of transformers in this domain. We provide comprehensive analysis of a vanilla transformer trained to perform n-digit integer multiplication. Our observations indicate that the model decomposes multiplication task into multiple parallel subtasks, sequentially optimizing each subtask for each digit to complete the final multiplication. Based on observation and analysis, we infer the reasons of transformers deficiencies in multiplication tasks lies in their difficulty in calculating successive carryovers and caching intermediate results, and confirmed this inference through experiments. Guided by these findings, we propose improvements to enhance transformers performance on multiplication tasks. These enhancements are validated through rigorous testing and mathematical modeling, not only enhance transformer's interpretability, but also improve its performance, e.g., we achieve over 99.9% accuracy on 5-digit integer multiplication with a tiny transformer, outperform LLMs GPT-4. Our method contributes to the broader fields of model understanding and interpretability, paving the way for analyzing more complex tasks and Transformer models. This work underscores the importance of explainable AI, helping to build trust in large language models and promoting their adoption in critical applications.

cs.CL↗

arXiv2024

Unsupervised Spike Depth Estimation via Cross-modality Cross-domain Knowledge Transfer

Neuromorphic spike data, an upcoming modality with high temporal resolution, has shown promising potential in autonomous driving by mitigating the challenges posed by high-velocity motion blur. However, training the spike depth estimation network holds significant challenges in two aspects: sparse spatial information for pixel-wise tasks and difficulties in achieving paired depth labels for temporally intensive spike streams. Therefore, we introduce open-source RGB data to support spike depth estimation, leveraging its annotations and spatial information. The inherent differences in modalities and data distribution make it challenging to directly apply transfer learning from open-source RGB to target spike data. To this end, we propose a cross-modality cross-domain (BiCross) framework to realize unsupervised spike depth estimation by introducing simulated mediate source spike data. Specifically, we design a Coarse-to-Fine Knowledge Distillation (CFKD) approach to facilitate comprehensive cross-modality knowledge transfer while preserving the unique strengths of both modalities, utilizing a spike-oriented uncertainty scheme. Then, we propose a Self-Correcting Teacher-Student (SCTS) mechanism to screen out reliable pixel-wise pseudo labels and ease the domain shift of the student model, which avoids error accumulation in target spike data. To verify the effectiveness of BiCross, we conduct extensive experiments on four scenarios, including Synthetic to Real, Extreme Weather, Scene Changing, and Real Spike. Our method achieves state-of-the-art (SOTA) performances, compared with RGB-oriented unsupervised depth estimation methods. Code and dataset: https://github.com/Theia-4869/BiCross

cs.CV↗

arXiv2024

Technique Report of CVPR 2024 PBDL Challenges

The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies. By leveraging the principles of physics to inform and enhance deep learning models, we can develop more robust and accurate vision systems. Physics-based vision aims to invert the processes to recover scene properties such as shape, reflectance, light distribution, and medium properties from images. In recent years, deep learning has shown promising improvements for various vision tasks, and when combined with physics-based vision, these approaches can enhance the robustness and accuracy of vision systems. This technical report summarizes the outcomes of the Physics-Based Vision Meets Deep Learning (PBDL) 2024 challenge, held in CVPR 2024 workshop. The challenge consisted of eight tracks, focusing on Low-Light Enhancement and Detection as well as High Dynamic Range (HDR) Imaging. This report details the objectives, methodologies, and results of each track, highlighting the top-performing solutions and their innovative approaches.

cs.CV↗

arXiv2024

Picosecond all-optical switching of Co/Gd based synthetic ferrimagnets

Single pulse all-optical switching of magnetization (AOS) in Co/Gd based synthetic ferrimagnets carries promises for hybrid spintronic-photonic integration. A crucial next step progressing towards this vision is to gain insight into AOS and multi-domain state (MDS) behavior using longer pulses, which is compatible with state-of-the-art integrated photonics. In this work, we present our studies on the AOS and MDS of [Co/Gd]n (n = 1, 2) using ps optical pulses across a large composition range. We theoretically and experimentally show that a large Gd layer thickness can enhance the AOS energy efficiency and maximum pulse duration. We have identified two augmenting roles of Gd in extending the maximum pulse duration. On the inter-atomic level, we found that more Gd offers a prolonged angular momentum supply to Co. On the micromagnetic level, a higher Gd content brings the system to be closer to magnetic compensation in the equilibrized hot state, thereby reducing the driving force for thermally assisted nucleation of domain walls, combating the formation of a MDS. Our study presents a composition overview of AOS in [Co/Gd]n and offers useful physical insights regarding AOS fundamentals as well as the projected photonic integration.

cond-mat.mes-hall↗

arXiv2024

XPose: eXplainable Human Pose Estimation

Current approaches in pose estimation primarily concentrate on enhancing model architectures, often overlooking the importance of comprehensively understanding the rationale behind model decisions. In this paper, we propose XPose, a novel framework that incorporates Explainable AI (XAI) principles into pose estimation. This integration aims to elucidate the individual contribution of each keypoint to final prediction, thereby elevating the model's transparency and interpretability. Conventional XAI techniques have predominantly addressed tasks with single-target tasks like classification. Additionally, the application of Shapley value, a common measure in XAI, to pose estimation has been hindered by prohibitive computational demands. To address these challenges, this work introduces an innovative concept called Group Shapley Value (GSV). This approach strategically organizes keypoints into clusters based on their interdependencies. Within these clusters, GSV meticulously calculates Shapley value for keypoints, while for inter-cluster keypoints, it opts for a more holistic group-level valuation. This dual-level computation framework meticulously assesses keypoint contributions to the final outcome, optimizing computational efficiency. Building on the insights into keypoint interactions, we devise a novel data augmentation technique known as Group-based Keypoint Removal (GKR). This method ingeniously removes individual keypoints during training phases, deliberately preserving those with strong mutual connections, thereby refining the model's predictive prowess for non-visible keypoints. The empirical validation of GKR across a spectrum of standard approaches attests to its efficacy. GKR's success demonstrates how using Explainable AI (XAI) can directly enhance pose estimation models.

cs.CV↗

arXiv2024

Rapidity scan approach for net-baryon cumulants with a statistical thermal model

Utilizing rapidity-dependent measurements to map the QCD phase diagram provides a complementary approach to traditional beam-energy-dependent measurements around midrapidity. The changing nature of thermodynamic properties of QCD matter along the beam axis in heavy-ion collisions at low collision energies both motivates and poses challenges for this method. In this study, we derive the analytical cumulant-generating function for subsystems within distinct rapidity windows, while accounting for global net-baryon charge conservation of the full system. Rapidity-dependent net-baryon cumulants are then calculated for a system exhibiting inhomogeneity along the beam axis, and their sensitivity to finite acceptances through changing rapidity bin widths is explored. We highlight the nontrivial behaviors exhibited by these cumulants, underscoring their importance in establishing a noncritical baseline for interpreting net-proton cumulants in the search for the QCD critical point. Finally, we discuss the implications of the rapidity scan for mapping the QCD phase diagram within the current context.

nucl-th↗

arXiv2024

Proximity QA: Unleashing the Power of Multi-Modal Large Language Models for Spatial Proximity Analysis

Multi-modal large language models (MLLMs) have demonstrated remarkable vision-language capabilities, primarily due to the exceptional in-context understanding and multi-task learning strengths of large language models (LLMs). The advent of visual instruction tuning has further enhanced MLLMs' performance in vision-language understanding. However, while existing MLLMs adeptly recognize \textit{what} objects are in an image, they still face challenges in effectively discerning \textit{where} these objects are, particularly along the distance (scene depth) axis. To overcome this limitation in MLLMs, we introduce Proximity Question Answering (Proximity QA), a novel framework designed to enable MLLMs to infer the proximity relationship between objects in images. The framework operates in two phases: the first phase focuses on guiding the models to understand the relative depth of objects, and the second phase further encourages the models to infer the proximity relationships between objects based on their depth perceptions. We also propose a VQA dataset called Proximity-110K, containing additional instructions that incorporate depth information and the proximity relationships of objects. We have conducted extensive experiments to validate Proximity QA's superior ability in depth perception and proximity analysis, outperforming other state-of-the-art MLLMs. Code and dataset will be released at \textcolor{magenta}{https://github.com/NorthSummer/ProximityQA.git}.

cs.CV↗

arXiv2024

Residual Alignment: Uncovering the Mechanisms of Residual Networks

The ResNet architecture has been widely adopted in deep learning due to its significant boost to performance through the use of simple skip connections, yet the underlying mechanisms leading to its success remain largely unknown. In this paper, we conduct a thorough empirical study of the ResNet architecture in classification tasks by linearizing its constituent residual blocks using Residual Jacobians and measuring their singular value decompositions. Our measurements reveal a process called Residual Alignment (RA) characterized by four properties: (RA1) intermediate representations of a given input are equispaced on a line, embedded in high dimensional space, as observed by Gai and Zhang [2021]; (RA2) top left and right singular vectors of Residual Jacobians align with each other and across different depths; (RA3) Residual Jacobians are at most rank C for fully-connected ResNets, where C is the number of classes; and (RA4) top singular values of Residual Jacobians scale inversely with depth. RA consistently occurs in models that generalize well, in both fully-connected and convolutional architectures, across various depths and widths, for varying numbers of classes, on all tested benchmark datasets, but ceases to occur once the skip connections are removed. It also provably occurs in a novel mathematical model we propose. This phenomenon reveals a strong alignment between residual branches of a ResNet (RA2+4), imparting a highly rigid geometric structure to the intermediate representations as they progress linearly through the network (RA1) up to the final layer, where they undergo Neural Collapse.

cs.LG↗

arXiv2023

VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows

Different from visible cameras which record intensity images frame by frame, the biologically inspired event camera produces a stream of asynchronous and sparse events with much lower latency. In practice, visible cameras can better perceive texture details and slow motion, while event cameras can be free from motion blurs and have a larger dynamic range which enables them to work well under fast motion and low illumination. Therefore, the two sensors can cooperate with each other to achieve more reliable object tracking. In this work, we propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios, and it is divided into a training and a testing subset, each of which contains 500 and 320 videos, respectively. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods by extending current single-modality trackers into dual-modality versions. More importantly, we further build a simple but effective tracking algorithm by proposing a cross-modality transformer, to achieve more effective feature fusion between visible and event data. Extensive experiments on the proposed VisEvent dataset, FE108, COESOT, and two simulated datasets (i.e., OTB-DVS and VOT-DVS), validated the effectiveness of our model. The dataset and source code have been released on: \url{https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark}.

cs.CV↗

arXiv2023

$U_A(1)$ symmetry restoration at high baryon density

We study the relation between chiral and $U_A(1)$ symmetries in the quark-meson model. Although quarks and mesons are described in mean field approximation, the topological susceptibility characterizing the $U_A(1)$ breaking comprises two components: one controlled by the condensate and the other by the meson fluctuation. The $U_A(1)$ restoration is governed by the competition of these components. In a hot medium, the condensates melt. However, the fluctuation is enhanced. Therefore, the $U_A(1)$ symmetry cannot be solely restored via the temperature effect. Nevertheless, the baryon density reduces the condensates and fluctuation, and thereby, the $U_A(1)$ symmetry can only be restored in a dense or dense and hot medium. The strange condensate plays a weak role in the susceptibility, and the chiral and $U_A(1)$ symmetry restorations occur almost at the same critical point.

hep-ph↗