arXiv subjects

Bohao Li

Publications and source records attributed to Bohao Li.

At least 19 recordsLinked to original sources

arXiv2026

AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

Detailed audio-video joint captioning is essential for multimodal video understanding and generation. However, prior works are constrained by three main limitations: (1) the scarcity of high-quality public datasets with fine-grained audio-visual joint captions; (2) reinforcement-learning methods that rely on coarse reward signals; and (3) the lack of a benchmark and metric for evaluating detailed audiovisual captions at the atomic level. To address these challenges, we propose: (1) AVCap-100K, a high-quality dataset of 100K temporally aligned, detail-rich audio-video captions; (2) AVCap, a model optimized via Detail-Aware GRPO (Da-GRPO) that achieves state-of-the-art performance among open-source models and matches or surpasses proprietary models on several evaluations; and (3) AVCap-Bench and AVCap-Score, a specialized benchmark and metric for evaluating atomic-level details in audiovisual captions. Our code, models, and datasets are available at https://huggingface.co/collections/Apryle/avcap.

cs.CV↗

arXiv2026

Twins: Learn to Predict Unified Representations with Focal Loss

Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations -- semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis -- resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe optimization imbalance: the model fits the ViT component well but struggles to match the VAE latent distribution. We trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimensionality, and condition-aligned vs condition-independent uncertainty. To address it, we adapt a focal regression objective for flow matching that upweights large-error VAE dimensions, better balancing optimization across the ViT and VAE components. On ImageNet, this yields up to 10.57 gFID gain over naive MSE loss without classifier-free guidance. Twins also performs competitively on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding- and generation-oriented representations.

cs.CV↗

arXiv2026

Text-Aided Multi-Modal Panoptic Symbol Spotting for CAD Floor Plan Drawings

Computer-Aided Design (CAD) floor plan drawings contain both graphical primitives and textual annotations, which provide complementary geometric and semantic cues for intelligent design understanding. Among CAD analysis tasks, panoptic symbol spotting has become increasingly important with the growing demand for industrial digitalization and deep learning-based automation. However, most existing methods remain primarily primitive-centric and underexploit textual annotations, despite their critical semantic value. Even the few text-aware approaches often treat annotations only superficially, without properly modeling complex syntax and hierarchical semantics of CAD annotations, which leads to semantic loss and suboptimal spotting performance. To address these limitations, we propose TextCAD, a multimodal framework that jointly models graphical primitives and textual annotations for panoptic symbol spotting. Specifically, we design a Type-Attribute Correlation Encoder (TACE) to explicitly encode the compositional semantics within annotations by jointly modeling their types and attributes. We further introduce a Semantic Hierarchy Alignment framework with Multi-level Semantic Filtering (MSF) and primitive downsampling, which adaptively aligns annotation semantics with graphical primitives at different semantic levels and enables accurate cross-modal semantic injection and fusion. Experiments on real-world building-design datasets show that TextCAD effectively improves symbol spotting performance and achieves state-of-the-art results.

cs.CV↗

arXiv2026

Quantum-Geometric Design of Lattice Generalized Landau Levels

We design lattice models with tailored quantum geometry, including generalized Landau levels (LLs) satisfying the integrated trace condition and higher-Chern bands with ideal quantum geometry. Our models with $N=2$, $3$, and $4$ sublattices include a generalized Haldane model ($N=2$ honeycomb lattice model) with Gaussian-decaying hoppings realizable in twisted bilayer MoTe$_2$, and $N \geq 3$ models with exponentially decaying hoppings. Exact diagonalization reveals fractional Chern insulators in the generalized zeroth LL bands of all three models, a Moore-Read state in the generalized first LL band of the $N=4$ model, and various interaction-driven topological phases$\unicode{x2013}$including integer and fractional anomalous Hall crystals and a multicomponent Halperin state$\unicode{x2013}$in the ideal higher-Chern band of the $N=3$ model. Informed by quantum geometry, our work provides a pathway for lattice realizations of Landau-level and beyond-Landau-level physics.

cond-mat.mes-hall↗

arXiv2026

UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.

cs.CL↗

arXiv2026

Simultaneous nanoscale imaging of local conductivity and chemical potential in a quantum Hall isospin ferromagnet

Quantum Hall isospin ferromagnetism in multilayer graphene offers a versatile playground for exploring flat band correlated physics, driven by the intricate coupling of spin, valley, orbital, and layer degrees of freedom. However, a nanoscale probe capable of simultaneously mapping local conductivity and chemical potential in these exotic phases has yet to be realized. Here, we introduce scanning conductivity and chemical potential microscopy (SCCM), a technique integrating scanning microwave impedance microscopy and Kelvin probe force microscopy. We demonstrate SCCM by probing the quantum Hall states and many-body Landau level energy spectrum in bilayer graphene. Applied to marginally twisted double bilayer graphene, SCCM then reveals a cascade of quantum Hall isospin ferromagnetic states with unexpected re-emergence behaviors. Significantly, experimental many-body Landau level energy spectrum further uncovers the intricate connections of these complex phenomena to inter-subband Landau level crossings and Landau level single-particle wavefunctions. These insights enable the construction of a comprehensive quantum Hall phase diagram. Our results demonstrate SCCM's capability in decoding complex quantum phenomena, establishing it as a versatile nanoscale probe for electron correlation and topology.

cond-mat.mes-hall↗

arXiv2026

A Clinical Point Cloud Paradigm for In-Hospital Mortality Prediction from Multi-Level Incomplete Multimodal EHRs

Deep learning-based modeling of multimodal Electronic Health Records (EHRs) has become an important approach for clinical diagnosis and risk prediction. However, due to diverse clinical workflows and privacy constraints, raw EHRs are inherently multi-level incomplete, including irregular sampling, missing modalities, and sparse labels. These issues cause temporal misalignment, modality imbalance, and limited supervision. Most existing multimodal methods assume relatively complete data, and even methods designed for incompleteness usually address only one or two of these issues in isolation. As a result, they often rely on rigid temporal/modal alignment or discard incomplete data, which may distort raw clinical semantics. To address this problem, we propose HealthPoint (HP), a unified clinical point cloud paradigm for multi-level incomplete EHRs. HP represents heterogeneous clinical events as points in a continuous 4D space defined by content, time, modality, and case. To model interactions between arbitrary point pairs, we introduce a Low-Rank Relational Attention mechanism that efficiently captures high-order dependencies across these four dimensions. We further develop a hierarchical interaction and sampling strategy to balance fine-grained modeling and computational efficiency. Built on this framework, HP enables flexible event-level interaction and fine-grained self-supervision, supporting robust modality recovery and effective use of unlabeled data. Experiments on large-scale EHR datasets for risk prediction show that HP consistently achieves state-of-the-art performance and strong robustness under varying degrees of incompleteness.

cs.LG↗

arXiv2026

CIGPose: Causal Intervention Graph Neural Network for Whole-Body Pose Estimation

State-of-the-art whole-body pose estimators often lack robustness, producing anatomically implausible predictions in challenging scenes. We posit this failure stems from spurious correlations learned from visual context, a problem we formalize using a Structural Causal Model (SCM). The SCM identifies visual context as a confounder that creates a non-causal backdoor path, corrupting the model's reasoning. We introduce the Causal Intervention Graph Pose (CIGPose) framework to address this by approximating the true causal effect between visual evidence and pose. The core of CIGPose is a novel Causal Intervention Module: it first identifies confounded keypoint representations via predictive uncertainty and then replaces them with learned, context-invariant canonical embeddings. These deconfounded embeddings are processed by a hierarchical graph neural network that reasons over the human skeleton at both local and global semantic levels to enforce anatomical plausibility. Extensive experiments show CIGPose achieves a new state-of-the-art on COCO-WholeBody. Notably, our CIGPose-x model achieves 67.0\% AP, surpassing prior methods that rely on extra training data. With the additional UBody dataset, CIGPose-x is further boosted to 67.5\% AP, demonstrating superior robustness and data efficiency. The codes and models are publicly available at https://github.com/53mins/CIGPose.

cs.CV↗

arXiv2026

Incident-Guided Spatiotemporal Traffic Forecasting

Recent years have witnessed the rapid development of deep-learning-based, graph-neural-network-based forecasting methods for modern intelligent transportation systems. However, most existing work focuses exclusively on capturing spatio-temporal dependencies from historical traffic data, while overlooking the fact that suddenly occurring transportation incidents, such as traffic accidents and adverse weather, serve as external disturbances that can substantially alter temporal patterns. We argue that this issue has become a major obstacle to modeling the dynamics of traffic systems and improving prediction accuracy, but the unpredictability of incidents makes it difficult to observe patterns from historical sequences. To address these challenges, this paper proposes a novel framework named the Incident-Guided Spatiotemporal Graph Neural Network (IGSTGNN). IGSTGNN explicitly models the incident's impact through two core components: an Incident-Context Spatial Fusion (ICSF) module to capture the initial heterogeneous spatial influence, and a Temporal Incident Impact Decay (TIID) module to model the subsequent dynamic dissipation. To facilitate research on the spatio-temporal impact of incidents on traffic flow, a large-scale dataset is constructed and released, featuring incident records that are time-aligned with traffic time series. On this new benchmark, the proposed IGSTGNN framework is demonstrated to achieve state-of-the-art performance. Furthermore, the generalizability of the ICSF and TIID modules is validated by integrating them into various existing models.

cs.LG↗

arXiv2026

Abelian and non-Abelian fractionalized states in twisted MoTe$_2$: A generalized Landau-level theory

Fractional Chern insulators are lattice analogs of fractional quantum Hall states that realize fractionalized quasiparticles without an external magnetic field. A key strategy to understand and design these phases is to map Chern bands onto Landau levels (LLs). Here, we introduce a universal framework that variationally decomposes Bloch bands into generalized LLs, providing a controlled and quantitative characterization of their effective LL nature. Applying this approach to twisted bilayer MoTe$_2$ modeled by first-principles-derived moir\'e Hamiltonians, we find that the first moir\'e valence band is dominated by the generalized zeroth LL across a broad range of twist angles, facilitating the formation of Abelian fractional Chern insulators in the Jain sequences. The second moir\'e band, renormalized via Hartree-Fock calculations at hole filling $\nu_h = 2$, is dominated by the generalized first LL at twist angles $\theta = 2.45^\circ$ and $2.13^\circ$. At $\theta = 2.45^\circ$, we find numerical evidence for a non-Abelian Moore--Read (MR) state at $\nu_h = 5/2$, with consistent signatures in both the energy spectrum and the particle entanglement spectrum. Interpolation studies further demonstrate an adiabatic connection between this state and the MR state in the conventional first LL. In contrast, at $\theta = 2.13^\circ$, a charge-density-wave state prevails in the competition with the MR state due to the larger bandwidth. Our variational mapping provides a theoretical framework for exploring exotic fractionalized phases, including non-Abelian states, in realistic systems.

cond-mat.mes-hall↗

arXiv2025

Text-Enhanced Panoptic Symbol Spotting in CAD Drawings

With the widespread adoption of Computer-Aided Design(CAD) drawings in engineering, architecture, and industrial design, the ability to accurately interpret and analyze these drawings has become increasingly critical. Among various subtasks, panoptic symbol spotting plays a vital role in enabling downstream applications such as CAD automation and design retrieval. Existing methods primarily focus on geometric primitives within the CAD drawings to address this task, but they face following major problems: they usually overlook the rich textual annotations present in CAD drawings and they lack explicit modeling of relationships among primitives, resulting in incomprehensive understanding of the holistic drawings. To fill this gap, we propose a panoptic symbol spotting framework that incorporates textual annotations. The framework constructs unified representations by jointly modeling geometric and textual primitives. Then, using visual features extract by pretrained CNN as the initial representations, a Transformer-based backbone is employed, enhanced with a type-aware attention mechanism to explicitly model the different types of spatial dependencies between various primitives. Extensive experiments on the real-world dataset demonstrate that the proposed method outperforms existing approaches on symbol spotting tasks involving textual annotations, and exhibits superior robustness when applied to complex CAD drawings.

cs.CV↗

arXiv2025

Quantum Phases in Twisted Homobilayer Transition Metal Dichalcogenides

Twisted homobilayer transition metal dichalcogenides - specifically twisted bilayer MoTe$_2$ and twisted bilayer WSe$_2$ - have recently emerged as a versatile platform for strongly correlated and topological phases of matter. These two-dimensional systems host tunable flat Chern bands in which Coulomb interactions can dominate over kinetic energy, giving rise to a variety of interaction-driven phenomena. A series of groundbreaking experiments have revealed a rich landscape of quantum phases, including integer and fractional quantum anomalous Hall states, quantum spin Hall states, anomalous Hall metals, zero-field composite Fermi liquids, and unconventional superconductors, along with more conventional topologically trivial correlated states including antiferromagnets. This review surveys recent experimental discoveries and theoretical progress in understanding these phases, with a focus on the key underlying mechanisms - band topology, electron interactions, symmetry breaking, and charge fractionalization. We emphasize the unique physics of twisted TMD homobilayers in comparison to other related systems, discuss open questions, and outline promising directions for future research.

cond-mat.str-el↗

arXiv2025

Characterization of fractional Chern insulator quasiparticles in twisted homobilayer MoTe$_2$

We provide a detailed study of Abelian quasiparticles of valley polarized fractional Chern insulators (FCIs) residing in the top valence band of twisted bilayer MoTe$_2$ (tMoTe$_2$) at hole filling $\nu_h=2/3$. We construct a tight-binding model of delocalized quasiparticles to capture the energy dispersion of a single quasiparticle. We then localize quasiparticles by short-range delta impurity potentials. Unlike the fractional quantum Hall (FQH) counterpart in the lowest Landau level (LLL), the density profile around the localized FCI quasiparticle in tMoTe$_2$ depends on the location of the impurity potential and loses the continuous rotation invariance. The FCI quasiparticle localized at moir\'e lattice center closely follows the anyon Wannier state of the tight-binding model of the mobile quasiparticle. Despite of the difference in density profiles, we find that the excess charge around the impurity potential for the $\nu_h=2/3$ FCIs in tMoTe$_2$ is still similar to that of the $\nu=2/3$ FQH state in the LLL if an effective magnetic length on the moir\'e lattice is chosen as the length unit, which allows a rough estimation of the spatial extent of the FCI quasiparticle. Far away from the impurity potential, this excess charge has the tendency to reach $e/3$, as expected for the Laughlin quasiparticle. The braiding phase of two FCI quasiparticles in tMoTe$_2$ also agrees with the theoretical prediction of fractional statistics. We characterize the interaction between two FCI quasiparticles and find a crossover from repulsive to attractive interaction as gate-to-sample distances decreases. Based on the nearly ideal quantum geometry of the top valence band of tMoTe$_2$, we propose a trial wave function for localized FCI quasiparticles, which reproduces the key feature of the density profile around a quasiparticle.

cond-mat.str-el↗

arXiv2025

Video-R1: Reinforcing Video Reasoning in MLLMs

Inspired by DeepSeek-R1's success in eliciting reasoning abilities through rule-based reinforcement learning (RL), we introduce Video-R1 as the first attempt to systematically explore the R1 paradigm for incentivizing video reasoning within multimodal large language models (MLLMs). However, directly applying RL training with the GRPO algorithm to video reasoning presents two primary challenges: (i) a lack of temporal modeling for video reasoning, and (ii) the scarcity of high-quality video-reasoning data. To address these issues, we first propose the T-GRPO algorithm, which encourages models to utilize temporal information in videos for reasoning. Additionally, instead of relying solely on video data, we incorporate high-quality image-reasoning data into the training process. We have constructed two datasets: Video-R1-CoT-165k for SFT cold start and Video-R1-260k for RL training, both comprising image and video data. Experimental results demonstrate that Video-R1 achieves significant improvements on video reasoning benchmarks such as VideoMMMU and VSI-Bench, as well as on general video benchmarks including MVBench and TempCompass, etc. Notably, Video-R1-7B attains a 37.1% accuracy on video spatial reasoning benchmark VSI-bench, surpassing the commercial proprietary model GPT-4o. All code, models, and data are released in: https://github.com/tulerfeng/Video-R1.

cs.CV↗

arXiv2025

Deep Learning Sheds Light on Integer and Fractional Topological Insulators

Electronic topological phases of matter, characterized by robust boundary states derived from topologically nontrivial bulk states, are pivotal for next-generation electronic devices. However, understanding their complex quantum phases, especially at larger scales and fractional fillings with strong electron correlations, has long posed a formidable computational challenge. Here, we employ a deep learning framework to express the many-body wavefunction of topological states in twisted ${\rm MoTe_2}$ systems, where diverse topological states are observed. Leveraging neural networks, we demonstrate the ability to identify and characterize topological phases, including the integer and fractional Chern insulators as well as the $Z_2$ topological insulators. Our deep learning approach significantly outperforms traditional methods, not only in computational efficiency but also in accuracy, enabling us to study larger systems and differentiate between competing phases such as fractional Chern insulators and charge density waves. Our predictions align closely with experimental observations, highlighting the potential of deep learning techniques to explore the rich landscape of topological and strongly correlated phenomena.

cond-mat.str-el↗

arXiv2025

Visualization of intervalley coherent phase in PtSe2/HOPG heterojunction

Intervalley coherent (IVC) phase in graphene systems arises from the coherent superposition of wave functions of opposite valleys, whose direct microscopic visualization provides pivotal insight into the emergent physics but remains elusive. Here, we successfully visualize the IVC phase in a heterostructure of monolayer PtSe2 on highly oriented pyrolytic graphite. Using spectroscopic imaging scanning tunneling microscopy, we observe a Root3 by Root3 modulation pattern superimposed on the higher-order moire superlattice of the heterostructure, which correlates with a small gap opening around the Fermi level and displays an anti-phase real-space conductance distribution of the two gap edges. Such modulation pattern and small-gap vanish on the heterostructure of monolayer PtSe2 on bilayer-graphene-covered SiC substrate, due to the increased carrier density in the bilayer graphene. We provide a theoretical mechanism that the Root3 by Root3 modulation pattern originates from the IVC phase of few-layer graphene, which is magnified by the higher-order moire superlattice. Our work achieves visualization of the IVC phase, and develops an avenue for its generation and amplification via a moir\'e interface.

cond-mat.mes-hall↗

arXiv2024

AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?

Recently, multimodal large language models (MLLMs), such as GPT-4o, Gemini 1.5 Pro, and Reka Core, have expanded their capabilities to include vision and audio modalities. While these models demonstrate impressive performance across a wide range of audio-visual applications, our proposed DeafTest reveals that MLLMs often struggle with simple tasks humans find trivial: 1) determining which of two sounds is louder, and 2) determining which of two sounds has a higher pitch. Motivated by these observations, we introduce AV-Odyssey Bench, a comprehensive audio-visual benchmark designed to assess whether those MLLMs can truly understand the audio-visual information. This benchmark encompasses 4,555 carefully crafted problems, each incorporating text, visual, and audio components. To successfully infer answers, models must effectively leverage clues from both visual and audio inputs. To ensure precise and objective evaluation of MLLM responses, we have structured the questions as multiple-choice, eliminating the need for human evaluation or LLM-assisted assessment. We benchmark a series of closed-source and open-source models and summarize the observations. By revealing the limitations of current models, we aim to provide useful insight for future dataset collection and model development.

cs.CV↗

arXiv2024

VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI

Recent advancements in Multi-modal Large Language Models (MLLMs) have opened new avenues for applications in Embodied AI. Building on previous work, EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating egocentric video understanding capabilities. To bridge the gap between MLLMs and low-level control in Embodied AI, we design four key interrelated tasks: video question-answering, hierarchy planning, visual grounding and reward modeling. To minimize manual annotation costs, we develop an automatic data generation pipeline based on the Ego4D dataset, leveraging the prior knowledge and multimodal capabilities of GPT-4o. Three human annotators then filter the generated data to ensure diversity and quality, resulting in the VidEgoThink benchmark. We conduct extensive experiments with three types of models: API-based MLLMs, open-source image-based MLLMs, and open-source video-based MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform poorly across all tasks related to egocentric video understanding. These findings suggest that foundation models still require significant advancements to be effectively applied to first-person scenarios in Embodied AI. In conclusion, VidEgoThink reflects a research trend towards employing MLLMs for egocentric vision, akin to human capabilities, enabling active observation and interaction in the complex real-world environments.

cs.CV↗