Searcharxiv⌕ Search

arXiv subjects

Bo Zhao

Publications and source records attributed to Bo Zhao.

At least 199 records · Page 11Linked to original sources
arXiv2024

M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models

Medical image analysis is essential to clinical diagnosis and treatment, which is increasingly supported by multi-modal large language models (MLLMs). However, previous research has primarily focused on 2D medical images, leaving 3D images under-explored, despite their richer spatial information. This paper aims to advance 3D medical image analysis with MLLMs. To this end, we present a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K image-text pairs and 662K instruction-response pairs specifically tailored for various 3D medical tasks, such as image-text retrieval, report generation, visual question answering, positioning, and segmentation. Additionally, we propose M3D-LaMed, a versatile multi-modal large language model for 3D medical image analysis. Furthermore, we introduce a new 3D multi-modal medical benchmark, M3D-Bench, which facilitates automatic evaluation across eight tasks. Through comprehensive evaluation, our method proves to be a robust model for 3D medical image analysis, outperforming existing solutions. All code, data, and models are publicly available at: https://github.com/BAAI-DCAI/M3D.

cs.CV↗
arXiv2024

Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability

Auto-regressive models have achieved impressive results in 2D image generation by modeling joint distributions in grid space. In this paper, we extend auto-regressive models to 3D domains, and seek a stronger ability of 3D shape generation by improving auto-regressive models at capacity and scalability simultaneously. Firstly, we leverage an ensemble of publicly available 3D datasets to facilitate the training of large-scale models. It consists of a comprehensive collection of approximately 900,000 objects, with multiple properties of meshes, points, voxels, rendered images, and text captions. This diverse labeled dataset, termed Objaverse-Mix, empowers our model to learn from a wide range of object variations. However, directly applying 3D auto-regression encounters critical challenges of high computational demands on volumetric grids and ambiguous auto-regressive order along grid dimensions, resulting in inferior quality of 3D shapes. To this end, we then present a novel framework Argus3D in terms of capacity. Concretely, our approach introduces discrete representation learning based on a latent vector instead of volumetric grids, which not only reduces computational costs but also preserves essential geometric details by learning the joint distributions in a more tractable order. The capacity of conditional generation can thus be realized by simply concatenating various conditioning inputs to the latent vector, such as point clouds, categories, images, and texts. In addition, thanks to the simplicity of our model architecture, we naturally scale up our approach to a larger model with an impressive 3.6 billion parameters, further enhancing the quality of versatile 3D generation. Extensive experiments on four generation tasks demonstrate that Argus3D can synthesize diverse and faithful shapes across multiple categories, achieving remarkable performance.

cs.CV↗
arXiv2024

Real-Fake: Effective Training Data Synthesis Through Distribution Matching

Synthetic training data has gained prominence in numerous learning tasks and scenarios, offering advantages such as dataset augmentation, generalization evaluation, and privacy preservation. Despite these benefits, the efficiency of synthetic data generated by current methodologies remains inferior when training advanced deep models exclusively, limiting its practical utility. To address this challenge, we analyze the principles underlying training data synthesis for supervised learning and elucidate a principled theoretical framework from the distribution-matching perspective that explicates the mechanisms governing synthesis efficacy. Through extensive experiments, we demonstrate the effectiveness of our synthetic data across diverse image classification tasks, both as a replacement for and augmentation to real datasets, while also benefits such as out-of-distribution generalization, privacy preservation, and scalability. Specifically, we achieve 70.9% top1 classification accuracy on ImageNet1K when training solely with synthetic data equivalent to 1 X the original real data size, which increases to 76.0% when scaling up to 10 X synthetic data.

cs.LG↗
arXiv2024

Efficient size-prescribed $k$-core search

$k$-core is a subgraph where every node has at least $k$ neighbors within the subgraph. The $k$-core subgraphs has been employed in large platforms like Network Repository to comprehend the underlying structures and dynamics of the network. Existing studies have primarily focused on finding $k$-core groups without considering their size, despite the relevance of solution sizes in many real-world scenarios. This paper addresses this gap by introducing the size-prescribed $k$-core search (SPCS) problem, where the goal is to find a subgraph of a specified size that has the highest possible core number. We propose two algorithms, namely the {\it TSizeKcore-BU} and the {\it TSizeKcore-TD}, to identify cohesive subgraphs that satisfy both the $k$-core requirement and the size constraint. Our experimental results demonstrate the superiority of our approach in terms of solution quality and efficiency. The {\it TSizeKcore-BU} algorithm proves to be highly efficient in finding size-prescribed $k$-core subgraphs on large datasets, making it a favorable choice for such scenarios. On the other hand, the {\it TSizeKcore-TD} algorithm is better suited for small datasets where running time is less critical.

cs.DS↗
arXiv2024

Spin: An Efficient Secure Computation Framework with GPU Acceleration

Accuracy and efficiency remain challenges for multi-party computation (MPC) frameworks. Spin is a GPU-accelerated MPC framework that supports multiple computation parties and a dishonest majority adversarial setup. We propose optimized protocols for non-linear functions that are critical for machine learning, as well as several novel optimizations specific to attention that is the fundamental unit of Transformer models, allowing Spin to perform non-trivial CNNs training and Transformer inference without sacrificing security. At the backend level, Spin leverages GPU, CPU, and RDMA-enabled smart network cards for acceleration. Comprehensive evaluations demonstrate that Spin can be up to $2\times$ faster than the state-of-the-art for deep neural network training. For inference on a Transformer model with 18.9 million parameters, our attention-specific optimizations enable Spin to achieve better efficiency, less communication, and better accuracy.

cs.CR↗
arXiv2024

Controlling The Contrast Between Absorptivity and Emissivity in Nonreciprocal Thermal Emitters

Recent advancements in nonreciprocal thermal emitters challenge the conventional Kirchhoff's law, which states that emissivity and absorptivity should be equal for a given direction, frequency, and polarization. These emitters can break Kirchhoff's law and enable unprecedented thermal photon control capabilities. However, current studies mainly focus on increasing the magnitude of the contrast between emissivity and absorptivity, with little attention paid to how the sign or bandwidth of the contrast may be controlled. In this work, we show such control ability can be achieved by coupling resonances that can provide opposite contrasts between emissivity and absorptivity.

physics.optics↗
arXiv2024

Learning Position-Aware Implicit Neural Network for Real-World Face Inpainting

Face inpainting requires the model to have a precise global understanding of the facial position structure. Benefiting from the powerful capabilities of deep learning backbones, recent works in face inpainting have achieved decent performance in ideal setting (square shape with $512px$). However, existing methods often produce a visually unpleasant result, especially in the position-sensitive details (e.g., eyes and nose), when directly applied to arbitrary-shaped images in real-world scenarios. The visually unpleasant position-sensitive details indicate the shortcomings of existing methods in terms of position information processing capability. In this paper, we propose an \textbf{I}mplicit \textbf{N}eural \textbf{I}npainting \textbf{N}etwork (IN$^2$) to handle arbitrary-shape face images in real-world scenarios by explicit modeling for position information. Specifically, a downsample processing encoder is proposed to reduce information loss while obtaining the global semantic feature. A neighbor hybrid attention block is proposed with a hybrid attention mechanism to improve the facial understanding ability of the model without restricting the shape of the input. Finally, an implicit neural pyramid decoder is introduced to explicitly model position information and bridge the gap between low-resolution features and high-resolution output. Extensive experiments demonstrate the superiority of the proposed method in real-world face inpainting task.

cs.CV↗
arXiv2023

SVIT: Scaling up Visual Instruction Tuning

Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Although existing multimodal models present impressive performance of visual understanding and reasoning, their limits are still largely under-explored due to the scarcity of high-quality instruction tuning data. To push the limits of multimodal capability, we Scale up Visual Instruction Tuning (SVIT) by constructing a dataset of 4.2 million visual instruction tuning data including 1.6M conversation question-answer (QA) pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs and 106K detailed image descriptions. Besides the volume, the proposed dataset is also featured by the high quality and rich diversity, which is generated by prompting GPT-4 with the abundant manual annotations of images. We also propose a new data recipe to select subset with better diversity and balance, which evokes model's superior capabilities. Extensive experiments verify that SVIT-v1.5, trained on the proposed dataset, outperforms state-of-the-art Multimodal Large Language Models on popular benchmarks. The data and code are publicly available at https://github.com/BAAI-DCAI/Visual-Instruction-Tuning.

cs.CV↗
arXiv2023

Open-DDVM: A Reproduction and Extension of Diffusion Model for Optical Flow Estimation

Recently, Google proposes DDVM which for the first time demonstrates that a general diffusion model for image-to-image translation task works impressively well on optical flow estimation task without any specific designs like RAFT. However, DDVM is still a closed-source model with the expensive and private Palette-style pretraining. In this technical report, we present the first open-source DDVM by reproducing it. We study several design choices and find those important ones. By training on 40k public data with 4 GPUs, our reproduction achieves comparable performance to the closed-source DDVM. The code and model have been released in https://github.com/DQiaole/FlowDiffusion_pytorch.

cs.CV↗
arXiv2023

Structured Light Modal Interface via Liquid-Crystal Planar Optics

Recent advances in planar optics with geometric-phase superstructures have brought a new paradigm in the control of structured light and, in particular, has substantially enhanced the capabilities of generating and detecting orbital angular momentum (OAM) states of light and associated spatial modes. However, the structured modal interface that can reciprocally link OAM states via adiabatic control and access-associated higher-order geometric phase remains absent in planar optics. In this work, we propose and experimentally demonstrate a planar optical astigmatic retarder fabricated with liquid-crystal (LC) geometric phase. The LC superstructure was designed with the principle of fractional Fourier transformation and is capable of reciprocal conversion between all possible OAM states on the same modal sphere. Such a planar device paves the way towards an easily deployed modal interface of paraxial OAM states, unlocks the resource of higher-order geometric phase, and has promising applications in high-dimensional classical/quantum information.

physics.optics↗
arXiv2023

Exploring the dust grain size and polarization mechanism in the hot and massive Class 0 disk IRAS 16293-2422 B

Multiwavelength dust continuum and polarization observations arising from self-scattering have been used to investigate grain sizes in young disks. However, the polarization by self-scattering is low in face-on optically thick disks and puts some of the size constraints from polarization on hold, particularly for the younger and more massive disks. The 1.3 mm emission detected toward the hot ($\gtrsim$400 K) Class 0 disk IRAS 16293-2422 B has been attributed to self-scattering, predicting grain sizes between 200-2000 $μ$m. We investigate the effects of grain size in the resultant flux and polarization fractions from self-scattering using a hot and massive Class 0 disk model and compare with observations. We compared new and archival high-resolution observations between 1.3 and 18 mm to a set of synthetic models. We have developed a new public tool to automate this process called Synthesizer. This is an easy-to-use program to generate synthetic observations from numerical simulations. Optical depths are in the range of 130 to 2 from 1.3 to 18 mm, respectively. Predictions from significant grain growth populations, including millimetric grains are comparable to the observations at all wavelengths. The polarization fraction produced by self-scattering reaches a maximum of $\sim$0.1% at 1.3 mm for a maximum grain size of 100 $μ$m, being an order of magnitude lower than that observed with ALMA. From the comparison of Stokes I fluxes, we conclude that significant grain growth could be present in the young Class 0 disk IRAS 16293 B, particularly in the inner hot region ($<10$ au, $T>$ 300 K) where refractory organics evaporate. The polarization produced by self-scattering in our model is not high enough to explain the observations at 1.3 and 7 mm, and effects like dichroic extinction or polarization reversal of elongated aligned grains remain other possible but untested scenarios.

astro-ph.SR↗
arXiv2023

Single-shot, full characterization of the spatial wavefunction of light fields via Stokes tomography

Since the diffraction behavior of a light field is fully determined by its spatial wavefunction, i.e., its spatial complex amplitude (SCA), full characterization of spatial wavefunction, plays a vital role in modern optics from both the fundamental and applied aspects. In this work, we present a novel complex-amplitude profiler based on spatial Stokes tomography with the capability to fully determine the SCA of a light field in a single shot with high precision and resolution. The SCA slice observed at any propagation plane provides complete information about the light field, thus allowing us to further retrieve the complete beam structure in 3 dimensions space, as well as the exact modal constitution in terms of spatial degrees of freedom. The principle demonstrated here provides an important advancement for the full characterization of light beams with a broad spectrum of potential applications in various areas of optics, especially for the growing field of structured light.

physics.optics↗
arXiv2023

A global product of fine-scale urban building height based on spaceborne lidar

Characterizing urban environments with broad coverages and high precision is more important than ever for achieving the UN's Sustainable Development Goals (SDGs) as half of the world's populations are living in cities. Urban building height as a fundamental 3D urban structural feature has far-reaching applications. However, so far, producing readily available datasets of recent urban building heights with fine spatial resolutions and global coverages remains a challenging task. Here, we provide an up-to-date global product of urban building heights based on a fine grid size of 150 m around 2020 by combining the spaceborne lidar instrument of GEDI and multi-sourced data including remotely sensed images (i.e., Landsat-8, Sentinel-2, and Sentinel-1) and topographic data. Our results revealed that the estimated method of building height samples based on the GEDI data was effective with 0.78 of Pearson's r and 3.67 m of RMSE in comparison to the reference data. The mapping product also demonstrated good performance as indicated by its strong correlation with the reference data (i.e., Pearson's r = 0.71, RMSE = 4.60 m). Compared with the currently existing products, our global urban building height map holds the ability to provide a higher spatial resolution (i.e., 150 m) with a great level of inherent details about the spatial heterogeneity and flexibility of updating using the GEDI samples as inputs. This work will boost future urban studies across many fields including climate, environmental, ecological, and social sciences.

cs.LG↗
arXiv2023

DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting

While diffusion models can successfully generate data and make predictions, they are predominantly designed for static images. We propose an approach for efficiently training diffusion models for probabilistic spatiotemporal forecasting, where generating stable and accurate rollout forecasts remains challenging, Our method, DYffusion, leverages the temporal dynamics in the data, directly coupling it with the diffusion steps in the model. We train a stochastic, time-conditioned interpolator and a forecaster network that mimic the forward and reverse processes of standard diffusion models, respectively. DYffusion naturally facilitates multi-step and long-range forecasting, allowing for highly flexible, continuous-time sampling trajectories and the ability to trade-off performance with accelerated sampling at inference time. In addition, the dynamics-informed diffusion process in DYffusion imposes a strong inductive bias and significantly improves computational efficiency compared to traditional Gaussian noise-based diffusion models. Our approach performs competitively on probabilistic forecasting of complex dynamics in sea surface temperatures, Navier-Stokes flows, and spring mesh systems.

cs.LG↗
arXiv2023

Pushing the Limits of 3D Shape Generation at Scale

We present a significant breakthrough in 3D shape generation by scaling it to unprecedented dimensions. Through the adaptation of the Auto-Regressive model and the utilization of large language models, we have developed a remarkable model with an astounding 3.6 billion trainable parameters, establishing it as the largest 3D shape generation model to date, named Argus-3D. Our approach addresses the limitations of existing methods by enhancing the quality and diversity of generated 3D shapes. To tackle the challenges of high-resolution 3D shape generation, our model incorporates tri-plane features as latent representations, effectively reducing computational complexity. Additionally, we introduce a discrete codebook for efficient quantization of these representations. Leveraging the power of transformers, we enable multi-modal conditional generation, facilitating the production of diverse and visually impressive 3D shapes. To train our expansive model, we leverage an ensemble of publicly-available 3D datasets, consisting of a comprehensive collection of approximately 900,000 objects from renowned repositories such as ModelNet40, ShapeNet, Pix3D, 3D-Future, and Objaverse. This diverse dataset empowers our model to learn from a wide range of object variations, bolstering its ability to generate high-quality and diverse 3D shapes. Extensive experimentation demonstrate the remarkable efficacy of our approach in significantly improving the visual quality of generated 3D shapes. By pushing the boundaries of 3D generation, introducing novel methods for latent representation learning, and harnessing the power of transformers for multi-modal conditional generation, our contributions pave the way for substantial advancements in the field. Our work unlocks new possibilities for applications in gaming, virtual reality, product design, and other domains that demand high-quality and diverse 3D objects.

cs.CV↗
arXiv2023

Sensing the Pulse of the Pandemic: Geovisualizing the Demographic Disparities of Public Sentiment toward COVID-19 through Social Media

Social media offers a unique lens to observe large-scale, spatial-temporal patterns of users reactions toward critical events. However, social media use varies across demographics, with younger users being more prevalent compared to older populations. This difference introduces biases in data representativeness, and analysis based on social media without proper adjustment will lead to overlooking the voices of digitally marginalized communities and inaccurate estimations. This study explores solutions to pinpoint and alleviate the demographic biases in social media analysis through a case study estimating the public sentiment about COVID-19 using Twitter data. We analyzed the pandemic-related Twitter data in the U.S. during 2020-2021 to (1) elucidate the uneven social media usage among demographic groups and the disparities of their sentiments toward COVID-19, (2) construct an adjusted public sentiment measurement based on social media, the Sentiment Adjusted by Demographics (SAD) index, to evaluate the spatiotemporal varying public sentiment toward COVID-19. The results show higher proportions of female and adolescent Twitter users expressing negative emotions to COVID-19. The SAD index unveils that the public sentiment toward COVID-19 was most negative in January and February 2020 and most positive in April 2020. Vermont and Wyoming were the most positive and negative states toward COVID-19.

cs.CY↗
arXiv2023

Observation of photoassociation resonances in ultracold atom-molecule collisions

Photoassociation of ultracold atoms is a resonant light-assisted collision process, in which two colliding atoms absorb a photon and form an excited molecule. Since the first observation about three decades ago, the photoassociation of ultracold atoms has made a significant impact on the study of ultracold atoms and molecules. Extending the photoassociation of atoms to the photoassociation of atom-molecule pairs or molecule-molecule pairs will offer many new opportunities in the study of precision polyatomic molecular spectroscopy, formation of ultracold polyatomic molecules, and quantum control of molecular collisions and reactions. However, the high density of states and the photoexcitation of the collision complex by the trapping laser make photoassociation into well-defined quantum states of polyatomic molecules extremely difficult. Here we report on the observation of photoassociation resonances in ultracold collisions between $^{23}$Na$^{40}$K molecules and $^{40}$K atoms. We perform photoassociation in a long-wavelength optical dipole trap to form deeply bound triatomic molecules in the electronically excited states. The atom-molecule Feshbach resonance is used to enhance the free-bound Franck-Condon overlap. The photoassociation into well-defined quantum states of excited triatomic molecules is identified by observing resonantly enhanced loss features. These loss features depend on the polarization of the photoassociation lasers, allowing us to assign the rotational quantum numbers. The observation of ultracold atom-molecule photoassociation resonances paves the way toward preparing ground-state triatomic molecules, provides a new high-resolution spectroscopy technique for polyatomic molecules, and is also important to atom-molecule Feshbach resonances.

physics.atom-ph↗
arXiv2023

Improvement of image-type very-low-energy-electron-diffraction spin polarimeter

Spin- and angle-resolved photoemission spectroscopy (SARPES) with high efficiency and resolution plays a crucial role in exploring the fine spin-resolved band structures of quantum materials. Here we report the performance of SARPES instrument with a second-generation home-made multichannel very-low-energy-electron-diffraction (VLEED) spin polarimeter. Its energy and angular resolutions achieve 7.2 meV and 0.52°. We present the results of SARPES measurements of Bi(111) film to demonstrate its performance. Combined with the density functional theory (DFT) calculations, the spin polarization of the bulk states was confirmed from the spin-layer locking caused by the local inversion asymmetry. The surface states at binding energy of 0.77 eV are found with 1.0 {\pm} 0.11 spin polarization. The better resolutions and stability compared with the first-generation one provide a good platform to investigate the spin-polarized electronic states in materials.

physics.ins-det↗