arXiv subjects

Jiamin Wu

Publications and source records attributed to Jiamin Wu.

49 records · Page 3Linked to original sources

arXiv2025

Coca-Splat: Collaborative Optimization for Camera Parameters and 3D Gaussians

In this work, we introduce Coca-Splat, a novel approach to addressing the challenges of sparse view pose-free scene reconstruction and novel view synthesis (NVS) by jointly optimizing camera parameters with 3D Gaussians. Inspired by deformable DEtection TRansformer, we design separate queries for 3D Gaussians and camera parameters and update them layer by layer through deformable Transformer layers, enabling joint optimization in a single network. This design demonstrates better performance because to accurately render views that closely approximate ground-truth images relies on precise estimation of both 3D Gaussians and camera parameters. In such a design, the centers of 3D Gaussians are projected onto each view by camera parameters to get projected points, which are regarded as 2D reference points in deformable cross-attention. With camera-aware multi-view deformable cross-attention (CaMDFA), 3D Gaussians and camera parameters are intrinsically connected by sharing the 2D reference points. Additionally, 2D reference point determined rays (RayRef) defined from camera centers to the reference points assist in modeling relationship between 3D Gaussians and camera parameters through RQ-decomposition on an overdetermined system of equations derived from the rays, enhancing the relationship between 3D Gaussians and camera parameters. Extensive evaluation shows that our approach outperforms previous methods, both pose-required and pose-free, on RealEstate10K and ACID within the same pose-free setting.

cs.CV↗

arXiv2025

HumanMM: Global Human Motion Recovery from Multi-shot Videos

In this paper, we present a novel framework designed to reconstruct long-sequence 3D human motion in the world coordinates from in-the-wild videos with multiple shot transitions. Such long-sequence in-the-wild motions are highly valuable to applications such as motion generation and motion understanding, but are of great challenge to be recovered due to abrupt shot transitions, partial occlusions, and dynamic backgrounds presented in such videos. Existing methods primarily focus on single-shot videos, where continuity is maintained within a single camera view, or simplify multi-shot alignment in camera space only. In this work, we tackle the challenges by integrating an enhanced camera pose estimation with Human Motion Recovery (HMR) by incorporating a shot transition detector and a robust alignment module for accurate pose and orientation continuity across shots. By leveraging a custom motion integrator, we effectively mitigate the problem of foot sliding and ensure temporal consistency in human pose. Extensive evaluations on our created multi-shot dataset from public 3D human datasets demonstrate the robustness of our method in reconstructing realistic human motion in world coordinates.

cs.CV↗

arXiv2025

MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data

Brain decoding aims to reconstruct visual perception of human subject from fMRI signals, which is crucial for understanding brain's perception mechanisms. Existing methods are confined to the single-subject paradigm due to substantial brain variability, which leads to weak generalization across individuals and incurs high training costs, exacerbated by limited availability of fMRI data. To address these challenges, we propose MindAligner, an explicit functional alignment framework for cross-subject brain decoding from limited fMRI data. The proposed MindAligner enjoys several merits. First, we learn a Brain Transfer Matrix (BTM) that projects the brain signals of an arbitrary new subject to one of the known subjects, enabling seamless use of pre-trained decoding models. Second, to facilitate reliable BTM learning, a Brain Functional Alignment module is proposed to perform soft cross-subject brain alignment under different visual stimuli with a multi-level brain alignment loss, uncovering fine-grained functional correspondences with high interpretability. Experiments indicate that MindAligner not only outperforms existing methods in visual decoding under data-limited conditions, but also provides valuable neuroscience insights in cross-subject functional analysis. The code will be made publicly available.

cs.CV↗

arXiv2024

An Exploration of Parallel Imaging System for Very-low Field (50mT) MRI Scanner

Reducing the scanning time of very-low field (VLF) magnetic resonance imaging (MRI) scanners, commonly employed for stroke diagnosis, can enhance patient comfort and operational efficiency. The conventional parallel imaging (PI) technique for high-field MRI should be tailored to apply here, considering the differences in the direction of the main magnetic field and the presence of noise. A VLF-specific PI algorithm and phased-array coil are proposed, marking the first application of PI in VLF MRI. Reconstruction quality is enhanced by denoising undersampled k-space data using a linear-prediction based Kalman filter. Subsequently, the denoised k-space data are nonlinearly mapped from the original space onto a high-dimensional feature space, utilizing a polynomial feature mapping defined nonlinear frame. Frame parameters are calculated using auto-calibration signals (ACS) from the center k-space, and missing phase-encoding lines in the original space are estimated using acquired lines in the feature space. An 8-channel phased-array coil, designed for a vertical main magnetic field, is decoupled using geometric overlap and a low input impedance (LII) preamplifier. Healthy volunteer head imaging experiments using the proposed PI technique exhibit the lowest mean-squared-error (MSE) value and the highest peak-signal-to-noise (PSNR) and structural similarity index (SSIM) values compared to two widely used PI methods. The proposed PI technique enables the VLF MRI scanner to achieve similar image quality and a 72.5% improvement in signal-to-noise ratio (SNR) compared to fully sampled images while requiring less than 50% of the scan time. We present a PI technique tailored for VLF MRI scanner for the first time, along with potential research direction to achieve greater reduction factor.

physics.med-ph↗

arXiv2024

PNR: Physics-informed Neural Representation for high-resolution LFM reconstruction

Light field microscopy (LFM) has been widely utilized in various fields for its capability to efficiently capture high-resolution 3D scenes. Despite the rapid advancements in neural representations, there are few methods specifically tailored for microscopic scenes. Existing approaches often do not adequately address issues such as the loss of high-frequency information due to defocus and sample aberration, resulting in suboptimal performance. In addition, existing methods, including RLD, INR, and supervised U-Net, face challenges such as sensitivity to initial estimates, reliance on extensive labeled data, and low computational efficiency, all of which significantly diminish the practicality in complex biological scenarios. This paper introduces PNR (Physics-informed Neural Representation), a method for high-resolution LFM reconstruction that significantly enhances performance. Our method incorporates an unsupervised and explicit feature representation approach, resulting in a 6.1 dB improvement in PSNR than RLD. Additionally, our method employs a frequency-based training loss, enabling better recovery of high-frequency details, which leads to a reduction in LPIPS by at least half compared to SOTA methods (1.762 V.S. 3.646 of DINER). Moreover, PNR integrates a physics-informed aberration correction strategy that optimizes Zernike polynomial parameters during optimization, thereby reducing the information loss caused by aberrations and improving spatial resolution. These advancements make PNR a promising solution for long-term high-resolution biological imaging applications. Our code and dataset will be made publicly available.

eess.IV↗

arXiv2024

TeethDreamer: 3D Teeth Reconstruction from Five Intra-oral Photographs

Orthodontic treatment usually requires regular face-to-face examinations to monitor dental conditions of the patients. When in-person diagnosis is not feasible, an alternative is to utilize five intra-oral photographs for remote dental monitoring. However, it lacks of 3D information, and how to reconstruct 3D dental models from such sparse view photographs is a challenging problem. In this study, we propose a 3D teeth reconstruction framework, named TeethDreamer, aiming to restore the shape and position of the upper and lower teeth. Given five intra-oral photographs, our approach first leverages a large diffusion model's prior knowledge to generate novel multi-view images with known poses to address sparse inputs and then reconstructs high-quality 3D teeth models by neural surface reconstruction. To ensure the 3D consistency across generated views, we integrate a 3D-aware feature attention mechanism in the reverse diffusion process. Moreover, a geometry-aware normal loss is incorporated into the teeth reconstruction process to enhance geometry accuracy. Extensive experiments demonstrate the superiority of our method over current state-of-the-arts, giving the potential to monitor orthodontic treatment remotely. Our code is available at https://github.com/ShanghaiTech-IMPACT/TeethDreamer

cs.CV↗

arXiv2024

Multi-modal Attribute Prompting for Vision-Language Models

Pre-trained Vision-Language Models (VLMs), like CLIP, exhibit strong generalization ability to downstream tasks but struggle in few-shot scenarios. Existing prompting techniques primarily focus on global text and image representations, yet overlooking multi-modal attribute characteristics. This limitation hinders the model's ability to perceive fine-grained visual details and restricts its generalization ability to a broader range of unseen classes. To address this issue, we propose a Multi-modal Attribute Prompting method (MAP) by jointly exploring textual attribute prompting, visual attribute prompting, and attribute-level alignment. The proposed MAP enjoys several merits. First, we introduce learnable visual attribute prompts enhanced by textual attribute semantics to adaptively capture visual attributes for images from unknown categories, boosting fine-grained visual perception capabilities for CLIP. Second, the proposed attribute-level alignment complements the global alignment to enhance the robustness of cross-modal alignment for open-vocabulary objects. To our knowledge, this is the first work to establish cross-modal attribute-level alignment for CLIP-based few-shot adaptation. Extensive experimental results on 11 datasets demonstrate that our method performs favorably against state-of-the-art approaches.

cs.CV↗

arXiv2023

ChatCounselor: A Large Language Models for Mental Health Support

This paper presents ChatCounselor, a large language model (LLM) solution designed to provide mental health support. Unlike generic chatbots, ChatCounselor is distinguished by its foundation in real conversations between consulting clients and professional psychologists, enabling it to possess specialized knowledge and counseling skills in the field of psychology. The training dataset, Psych8k, was constructed from 260 in-depth interviews, each spanning an hour. To assess the quality of counseling responses, the counseling Bench was devised. Leveraging GPT-4 and meticulously crafted prompts based on seven metrics of psychological counseling assessment, the model underwent evaluation using a set of real-world counseling questions. Impressively, ChatCounselor surpasses existing open-source models in the counseling Bench and approaches the performance level of ChatGPT, showcasing the remarkable enhancement in model capability attained through high-quality domain-specific data.

cs.CL↗

arXiv2023

Leveraging Side Information for Ligand Conformation Generation using Diffusion-Based Approaches

Ligand molecule conformation generation is a critical challenge in drug discovery. Deep learning models have been developed to tackle this problem, particularly through the use of generative models in recent years. However, these models often generate conformations that lack meaningful structure and randomness due to the absence of essential side information. Examples of such side information include the chemical and geometric features of the target protein, ligand-target compound interactions, and ligand chemical properties. Without these constraints, the generated conformations may not be suitable for further selection and design of new drugs. To address this limitation, we propose a novel method for generating ligand conformations that leverage side information and incorporate flexible constraints into standard diffusion models. Drawing inspiration from the concept of message passing, we introduce ligand-target massage passing block, a mechanism that facilitates the exchange of information between target nodes and ligand nodes, thereby incorporating target node features. To capture non-covalent interactions, we introduce ligand-target compound inter and intra edges. To further improve the biological relevance of the generated conformations, we train energy models using scalar chemical features. These models guide the progress of the standard Denoising Diffusion Probabilistic Models, resulting in more biologically meaningful conformations. We evaluate the performance of SIDEGEN using the PDBBind-2020 dataset, comparing it against other methods. The results demonstrate improvements in both Aligned RMSD and Ligand RMSD evaluations. Specifically, our model outperforms GeoDiff (trained on PDBBind-2020) by 20% in terms of the median aligned RMSD metric.

q-bio.BM↗

arXiv2022

Dynamic-quenching of a single-photon avalanche photodetector using an adaptive resistive switch

One of the most common approaches for quenching single-photon avalanche diodes is to use a passive resistor in series with it. A drawback of this approach has been the limited recovery speed of the single-photon avalanche diodes. High resistance is needed to quench the avalanche, leading to slower recharging of the single-photon avalanche diodes depletion capacitor. We address this issue by replacing a fixed quenching resistor with a bias-dependent adaptive resistive switch. Reversible generation of metallic conduction enables switching between low and high resistance states under unipolar bias. As an example, using a Pt/Al2O3/Ag resistor with a commercial silicon single-photon avalanche diodes, we demonstrate avalanche pulse widths as small as ~30 ns, 10x smaller than a passively quenched approach, thus significantly improving the single-photon avalanche diodes frequency response. The experimental results are consistent with a model where the adaptive resistor dynamically changes its resistance during discharging and recharging the single-photon avalanche diodes.

physics.ins-det↗

arXiv2022

Photonic unsupervised learning processor for secure and high-throughput optical fiber communication

Following the explosive growth of global data, there is an ever-increasing demand for high-throughput optical fiber communication (OFC) systems to perform massive data transmission and processing. Existing OFC methods mainly rely on electronic circuits for data processing, which severely limits the communication throughput. Though considered promising for the next-generation high-speed fiber communication, all-optical OFC remains unachievable due to serious challenges in effective optical computing, system modeling and configuring. Here we propose an end-to-end photonic encoder-decoder (PED) processor which maps the physical system of OFC into an optical generative neural network. By modeling the OFC transmission process as the variation in the constructed optical latent space, the PED learns noise-resistant coding schemes via unsupervised optimization. With multi-layer parametric diffractive neural networks, the PED establishes a large-scale and high-throughput optical computing framework that integrates the main OFC computations including coding, encryption and compression to the optical domain. The whole system improves the latency of computation in OFC systems by five orders of magnitude compared with the state-of-the-art device. On benchmarking datasets, the PED experimentally achieves up to 32% reduction in transmission error ratio (ER) than on-off keying (OOK), one of the mainstream methods with the lowest ER in general transmission. As we demonstrate on medical data, the PED increases the transmission throughput by two orders of magnitude than 8-level pulse amplitude modulation (PAM-8). We believe the proposed photonic encoder-decoder processor not only paves the way to the next-generation all-optical OFC systems, but also promotes a wide range of AI-based physical system designs.

physics.optics↗

arXiv2020

Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit

Application-specific optical processors have been considered disruptive technologies for modern computing that can fundamentally accelerate the development of artificial intelligence (AI) by offering substantially improved computing performance. Recent advancements in optical neural network architectures for neural information processing have been applied to perform various machine learning tasks. However, the existing architectures have limited complexity and performance; and each of them requires its own dedicated design that cannot be reconfigured to switch between different neural network models for different applications after deployment. Here, we propose an optoelectronic reconfigurable computing paradigm by constructing a diffractive processing unit (DPU) that can efficiently support different neural networks and achieve a high model complexity with millions of neurons. It allocates almost all of its computational operations optically and achieves extremely high speed of data modulation and large-scale network parameter updating by dynamically programming optical modulators and photodetectors. We demonstrated the reconfiguration of the DPU to implement various diffractive feedforward and recurrent neural networks and developed a novel adaptive training approach to circumvent the system imperfections. We applied the trained networks for high-speed classifying of handwritten digit images and human action videos over benchmark datasets, and the experimental results revealed a comparable classification accuracy to the electronic computing approaches. Furthermore, our prototype system built with off-the-shelf optoelectronic components surpasses the performance of state-of-the-art graphics processing units (GPUs) by several times on computing speed and more than an order of magnitude on system energy efficiency.

eess.IV↗

arXiv2018

Fast 3D cell tracking with wide-field fluorescence microscopy through deep learning

Tracking cells in 3D at high speed continues to attract extensive attention for many biomedical applications, such as monitoring immune cell migration and observing tumor metastasis in flowing blood vessels. Here, we propose a deep convolutional neural networks (CNNs) based method to retrieve the 3D locations of the fluorophores from a single 2D image captured by a conventional wide-field fluorescence microscope without any hardware modification. The reported method converts the challenging 3D localization from an ill-posed model-based fitting problem, especially with dense samples and low signal-to-noise ratio, to a solvable multi-label classification problem through two cascaded CNNs, where deep learning technique has a great advantage over other algorithms. Compared with traditional kernel-fitting methods, the proposed method achieves more accurate and robust localization of multiple objects across a much larger axial range, which is validated by both simulation and experimental results on 3D distributed fluorescent beads. Moreover, in vivo 3D tracking of multiple blood cells in zebrafish at 100 fps further verifies the feasibility of our framework.

physics.optics↗