arXiv subjects

Zhao-Yang Wang

Publications and source records attributed to Zhao-Yang Wang.

11 recordsLinked to original sources

arXiv2026

Combo-Gait: Unified Transformer Framework for Multi-Modal Gait Recognition and Attribute Analysis

Gait recognition is an important biometric for human identification at a distance, particularly under low-resolution or unconstrained environments. Current works typically focus on either 2D representations (e.g., silhouettes and skeletons) or 3D representations (e.g., meshes and SMPLs), but relying on a single modality often fails to capture the full geometric and dynamic complexity of human walking patterns. In this paper, we propose a multi-modal and multi-task framework that combines 2D temporal silhouettes with 3D SMPL features for robust gait analysis. Beyond identification, we introduce a multitask learning strategy that jointly performs gait recognition and human attribute estimation, including age, body mass index (BMI), and gender. A unified transformer is employed to effectively fuse multi-modal gait features and better learn attribute-related representations, while preserving discriminative identity cues. Extensive experiments on the large-scale BRIAR datasets, collected under challenging conditions such as long-range distances (up to 1 km) and extreme pitch angles (up to 50°), demonstrate that our approach outperforms state-of-the-art methods in gait recognition and provides accurate human attribute estimation. These results highlight the promise of multi-modal and multitask learning for advancing gait-based human understanding in real-world scenarios.

cs.CV↗

arXiv2025

Mesh-Gait: A Unified Framework for Gait Recognition Through Multi-Modal Representation Learning from 2D Silhouettes

Gait recognition, a fundamental biometric technology, leverages unique walking patterns for individual identification, typically using 2D representations such as silhouettes or skeletons. However, these methods often struggle with viewpoint variations, occlusions, and noise. Multi-modal approaches that incorporate 3D body shape information offer improved robustness but are computationally expensive, limiting their feasibility for real-time applications. To address these challenges, we introduce Mesh-Gait, a novel end-to-end multi-modal gait recognition framework that directly reconstructs 3D representations from 2D silhouettes, effectively combining the strengths of both modalities. Compared to existing methods, directly learning 3D features from 3D joints or meshes is complex and difficult to fuse with silhouette-based gait features. To overcome this, Mesh-Gait reconstructs 3D heatmaps as an intermediate representation, enabling the model to effectively capture 3D geometric information while maintaining simplicity and computational efficiency. During training, the intermediate 3D heatmaps are gradually reconstructed and become increasingly accurate under supervised learning, where the loss is calculated between the reconstructed 3D joints, virtual markers, and 3D meshes and their corresponding ground truth, ensuring precise spatial alignment and consistent 3D structure. Mesh-Gait extracts discriminative features from both silhouettes and reconstructed 3D heatmaps in a computationally efficient manner. This design enables the model to capture spatial and structural gait characteristics while avoiding the heavy overhead of direct 3D reconstruction from RGB videos, allowing the network to focus on motion dynamics rather than irrelevant visual details. Extensive experiments demonstrate that Mesh-Gait achieves state-of-the-art accuracy. The code will be released upon acceptance of the paper.

cs.CV↗

arXiv2025

Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planning

Providing effective treatment and making informed clinical decisions are essential goals of modern medicine and clinical care. We are interested in simulating disease dynamics for clinical decision-making, leveraging recent advances in large generative models. To this end, we introduce the Medical World Model (MeWM), the first world model in medicine that visually predicts future disease states based on clinical decisions. MeWM comprises (i) vision-language models to serve as policy models, and (ii) tumor generative models as dynamics models. The policy model generates action plans, such as clinical treatments, while the dynamics model simulates tumor progression or regression under given treatment conditions. Building on this, we propose the inverse dynamics model that applies survival analysis to the simulated post-treatment tumor, enabling the evaluation of treatment efficacy and the selection of the optimal clinical action plan. As a result, the proposed MeWM simulates disease dynamics by synthesizing post-treatment tumors, with state-of-the-art specificity in Turing tests evaluated by radiologists. Simultaneously, its inverse dynamics model outperforms medical-specialized GPTs in optimizing individualized treatment protocols across all metrics. Notably, MeWM improves clinical decision-making for interventional physicians, boosting F1-score in selecting the optimal TACE protocol by 13%, paving the way for future integration of medical world models as the second readers.

cs.CV↗

arXiv2025

Investigating the light curve variation of magnetic white dwarfs induced by the axion-photon conversion

Axion-photon oscillation refers to the process of mutual conversion between photons and axions when they propagate in a magnetic field. This process depends on the strength of the background magnetic field, and magnetic white dwarfs provide a natural laboratory for testing this process. In this work, we study the behavior of axion-photon oscillation near magnetic white dwarfs: as the magnetic white dwarf rotates, its magnetic field structure rotates accordingly, causing a periodic change of the magnetic field along the path of photons. These variations affect the axion-photon oscillation process experienced by the photons emitted from the white dwarf, thereby inducing a periodic modulation in the intensity and polarization of the white dwarf's thermal emission that we observe. Our study focuses on the impact of axion effects on the observed light curve variation and conducts a detailed investigation through numerical calculations. Using the light curve data of the white dwarf PG1015+014 obtained from the observations by the Jacobus Kapteyn Telescope, which has a photometric precision of $\sim1\%$, we derive the constraints on axion parameters. In the axion mass range of $\lesssim10^{-8}\,{\rm eV}$, the 95\% credible interval upper limit of the axion-photon coupling $g_{aγγ}$ is constrained to $<8.1 \times 10^{-12} \mathrm{GeV^{-1}}$.

hep-ph↗

arXiv2024

An attempt to study axion-photon coupling using compact binary systems with high Shapiro time delay

We study the axion-photon conversion process in pulsar binary systems with high Shapiro time delay. In these binary systems, the orbital plane is nearly parallel to the line of sight. When the companion star is positioned between the pulsar and the Earth, the pulsar radiation beam will pass through the companion's magnetic field, potentially leading to axion-photon conversion that affects the intensity and linear polarization of the photons. The advantage of using such systems for axion or axion-like particle (ALP) research is that the intrinsic intensity and polarization state of the source photons can be well determined. This work analyzes the axion-photon conversion and the magnetic field along the photon path, establishing the corresponding model and solving it numerically. We choose PSR J1614-2230 and PSR J1910-5959A for our study. By assuming the companion's magnetic field, magnetic axis angle, and detector sensitivity, we discuss the feasibility of using high Shapiro delay binary systems to constrain the axion/ALP parameter space. We find that the upper limits derived from these two sources are weaker then existing constraints. However, considering the higher reliability, the method proposed in this work is still valuable. As more suitable samples and richer observational results are discovered, further constraints on the axion parameters could be strengthened.

astro-ph.HE↗

arXiv2024

Efficient Large Multi-modal Models via Visual Context Compression

While significant advancements have been made in compressed representations for text embeddings in large language models (LLMs), the compression of visual tokens in multi-modal LLMs (MLLMs) has remained a largely overlooked area. In this work, we present the study on the analysis of redundancy concerning visual tokens and efficient training within these models. Our initial experiments show that eliminating up to 70% of visual tokens at the testing stage by simply average pooling only leads to a minimal 3% reduction in visual question answering accuracy on the GQA benchmark, indicating significant redundancy in visual context. Addressing this, we introduce Visual Context Compressor, which reduces the number of visual tokens to enhance training and inference efficiency without sacrificing performance. To minimize information loss caused by the compression on visual tokens while maintaining training efficiency, we develop LLaVolta as a light and staged training scheme that incorporates stage-wise visual context compression to progressively compress the visual tokens from heavily to lightly compression during training, yielding no loss of information when testing. Extensive experiments demonstrate that our approach enhances the performance of MLLMs in both image-language and video-language understanding, while also significantly cutting training costs and improving inference efficiency.

cs.CV↗

arXiv2020

Z-Net: an Anisotropic 3D DCNN for Medical CT Volume Segmentation

Accurate volume segmentation from the Computed Tomography (CT) scan is a common prerequisite for pre-operative planning, intra-operative guidance and quantitative assessment of therapeutic outcomes in robot-assisted Minimally Invasive Surgery (MIS). 3D Deep Convolutional Neural Network (DCNN) is a viable solution for this task, but is memory intensive. Small isotropic patches are cropped from the original and large CT volume to mitigate this issue in practice, but it may cause discontinuities between the adjacent patches and severe class-imbalances within individual sub-volumes. This paper presents a new 3D DCNN framework, namely Z-Net, to tackle the discontinuity and class-imbalance issue by preserving a full field-of-view of the objects in the XY planes using anisotropic spatial separable convolutions. The proposed Z-Net can be seamlessly integrated into existing 3D DCNNs with isotropic convolutions such as 3D U-Net and V-Net, with improved volume segmentation Intersection over Union (IoU) - up to $12.6\%$. Detailed validation of Z-Net is provided for CT aortic, liver and lung segmentation, demonstrating the effectiveness and practical value of Z-Net for intra-operative 3D navigation in robot-assisted MIS.

eess.IV↗

arXiv2019

Instantiation-Net: 3D Mesh Reconstruction from Single 2D Image for Right Ventricle

3D shape instantiation which reconstructs the 3D shape of a target from limited 2D images or projections is an emerging technique for surgical intervention. It improves the currently less-informative and insufficient 2D navigation schemes for robot-assisted Minimally Invasive Surgery (MIS) to 3D navigation. Previously, a general and registration-free framework was proposed for 3D shape instantiation based on Kernel Partial Least Square Regression (KPLSR), requiring manually segmented anatomical structures as the pre-requisite. Two hyper-parameters including the Gaussian width and component number also need to be carefully adjusted. Deep Convolutional Neural Network (DCNN) based framework has also been proposed to reconstruct a 3D point cloud from a single 2D image, with end-to-end and fully automatic learning. In this paper, an Instantiation-Net is proposed to reconstruct the 3D mesh of a target from its a single 2D image, by using DCNN to extract features from the 2D image and Graph Convolutional Network (GCN) to reconstruct the 3D mesh, and using Fully Connected (FC) layers to connect the DCNN to GCN. Detailed validation was performed to demonstrate the practical strength of the method and its potential clinical use.

eess.IV↗

arXiv2019

U-Net Training with Instance-Layer Normalization

Normalization layers are essential in a Deep Convolutional Neural Network (DCNN). Various normalization methods have been proposed. The statistics used to normalize the feature maps can be computed at batch, channel, or instance level. However, in most of existing methods, the normalization for each layer is fixed. Batch-Instance Normalization (BIN) is one of the first proposed methods that combines two different normalization methods and achieve diverse normalization for different layers. However, two potential issues exist in BIN: first, the Clip function is not differentiable at input values of 0 and 1; second, the combined feature map is not with a normalized distribution which is harmful for signal propagation in DCNN. In this paper, an Instance-Layer Normalization (ILN) layer is proposed by using the Sigmoid function for the feature map combination, and cascading group normalization. The performance of ILN is validated on image segmentation of the Right Ventricle (RV) and Left Ventricle (LV) using U-Net as the network architecture. The results show that the proposed ILN outperforms previous traditional and popular normalization methods with noticeable accuracy improvements for most validations, supporting the effectiveness of the proposed ILN.

eess.IV↗

arXiv2019

One-stage Shape Instantiation from a Single 2D Image to 3D Point Cloud

Shape instantiation which predicts the 3D shape of a dynamic target from one or more 2D images is important for real-time intra-operative navigation. Previously, a general shape instantiation framework was proposed with manual image segmentation to generate a 2D Statistical Shape Model (SSM) and with Kernel Partial Least Square Regression (KPLSR) to learn the relationship between the 2D and 3D SSM for 3D shape prediction. In this paper, the two-stage shape instantiation is improved to be one-stage. PointOutNet with 19 convolutional layers and three fully-connected layers is used as the network structure and Chamfer distance is used as the loss function to predict the 3D target point cloud from a single 2D image. With the proposed one-stage shape instantiation algorithm, a spontaneous image-to-point cloud training and inference can be achieved. A dataset from 27 Right Ventricle (RV) subjects, indicating 609 experiments, were used to validate the proposed one-stage shape instantiation algorithm. An average point cloud-to-point cloud (PC-to-PC) error of 1.72mm has been achieved, which is comparable to the PLSR-based (1.42mm) and KPLSR-based (1.31mm) two-stage shape instantiation algorithm.

cs.CV↗

arXiv2011

Distinguishing manipulated stocks via trading network analysis

Manipulation is an important issue for both developed and emerging stock markets. For the study of manipulation, it is critical to analyze investor behavior in the stock market. In this paper, an analysis of the full transaction records of over a hundred stocks in a one-year period is conducted. For each stock, a trading network is constructed to characterize the relations among its investors. In trading networks, nodes represent investors and a directed link connects a stock seller to a buyer with the total trade size as the weight of the link, and the node strength is the sum of all edge weights of a node. For all these trading networks, we find that the node degree and node strength both have tails following a power-law distribution. Compared with non-manipulated stocks, manipulated stocks have a high lower bound of the power-law tail, a high average degree of the trading network and a low correlation between the price return and the seller-buyer ratio. These findings may help us to detect manipulated stocks.

q-fin.TR↗