Searcharxiv⌕ Search

arXiv subjects

Bo Zhang

Publications and source records attributed to Bo Zhang.

At least 433 records · Page 24Linked to original sources
arXiv2023

Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentation

Traditional 3D segmentation methods can only recognize a fixed range of classes that appear in the training set, which limits their application in real-world scenarios due to the lack of generalization ability. Large-scale visual-language pre-trained models, such as CLIP, have shown their generalization ability in the zero-shot 2D vision tasks, but are still unable to be applied to 3D semantic segmentation directly. In this work, we focus on zero-shot point cloud semantic segmentation and propose a simple yet effective baseline to transfer the visual-linguistic knowledge implied in CLIP to point cloud encoder at both feature and output levels. Both feature-level and output-level alignments are conducted between 2D and 3D encoders for effective knowledge transfer. Concretely, a Multi-granularity Cross-modal Feature Alignment (MCFA) module is proposed to align 2D and 3D features from global semantic and local position perspectives for feature-level alignment. For the output level, per-pixel pseudo labels of unseen classes are extracted using the pre-trained CLIP model as supervision for the 3D segmentation model to mimic the behavior of the CLIP image encoder. Extensive experiments are conducted on two popular benchmarks of point cloud segmentation. Our method outperforms significantly previous state-of-the-art methods under zero-shot setting (+29.2% mIoU on SemanticKITTI and 31.8% mIoU on nuScenes), and further achieves promising results in the annotation-free point cloud semantic segmentation setting, showing its great potential for label-efficient learning.

cs.CV↗
arXiv2023

Make RepVGG Greater Again: A Quantization-aware Approach

The tradeoff between performance and inference speed is critical for practical applications. Architecture reparameterization obtains better tradeoffs and it is becoming an increasingly popular ingredient in modern convolutional neural networks. Nonetheless, its quantization performance is usually too poor to deploy (more than 20% top-1 accuracy drop on ImageNet) when INT8 inference is desired. In this paper, we dive into the underlying mechanism of this failure, where the original design inevitably enlarges quantization error. We propose a simple, robust, and effective remedy to have a quantization-friendly structure that also enjoys reparameterization benefits. Our method greatly bridges the gap between INT8 and FP32 accuracy for RepVGG. Without bells and whistles, the top-1 accuracy drop on ImageNet is reduced within 2% by standard post-training quantization. Moreover, our method also achieves similar FP32 performance as RepVGG. Extensive experiments on detection and semantic segmentation tasks verify its generalization.

cs.CV↗
arXiv2023

Lenna: Language Enhanced Reasoning Detection Assistant

With the fast-paced development of multimodal large language models (MLLMs), we can now converse with AI systems in natural languages to understand images. However, the reasoning power and world knowledge embedded in the large language models have been much less investigated and exploited for image perception tasks. In this paper, we propose Lenna, a language-enhanced reasoning detection assistant, which utilizes the robust multimodal feature representation of MLLMs, while preserving location information for detection. This is achieved by incorporating an additional token in the MLLM vocabulary that is free of explicit semantic context but serves as a prompt for the detector to identify the corresponding position. To evaluate the reasoning capability of Lenna, we construct a ReasonDet dataset to measure its performance on reasoning-based detection. Remarkably, Lenna demonstrates outstanding performance on ReasonDet and comes with significantly low training costs. It also incurs minimal transferring overhead when extended to other tasks. Our code and model will be available at https://git.io/Lenna.

cs.CV↗
arXiv2023

Mössbauer spectroscopy study of the magnetostructural and spin-state transitions in the breathing pyrochlore LiFeCr$_{4}$O$_{8}$

We report on investigations of the complex magnetostructural and spin-state transitions in the breathing pyrochlore LiFeCr$_{4}$O$_{8}$ by means of magnetization, Mössbauer spectroscopy, and density functional theory (DFT) calculations. Three transitions corresponding to the ferrimagnetic transition at $T_N\sim94$ K, the spin-gap transition at $T_{SG}\sim50$ K, and the magnetostructural transition at $T_{MS}\sim19$ K were observed from the $χ$(T) curve, whereas only $T_N$ and $T_{MS}$ were evidenced for the Fe site from our Mössbauer measurements, suggesting that the spin-gap transition is absent at the Fe site. This indicates that the spin-gap transition is an effect of the breathing Cr$_4$ lattice, in agreement with our DFT calculations from which we see nearly decoupled electronic states for the FeO$_4$ and CrO$_6$ units. From the temperature dependence of the hyperfine magnetic field we also observed a spin-state transition for the Fe spins at $T_{MS}$ consistent with earlier neutron diffraction measurements. These local characteristics are believed to be important for a complete understanding of the complex magnetostructural coupling effects observed in similar systems.

cond-mat.str-el↗
arXiv2023

VLBI detection of the AE Aqr twin, LAMOST J024048.51+195226.9

LAMOST J024048.51+195226.9 (J0240+1952) was recently identified as the second AE Aquarii (AE Aqr)-type cataclysmic variable, possessing the fastest known rotating white dwarf. We performed a Very Long Baseline Interferometry (VLBI) observation of J0240+1952 utilizing the European VLBI Network at 1.7\,GHz, to obtain the first view of the radio morphology on mas scale. Our high-resolution VLBI image clearly shows that the radio emission is compact on mas scale ($\lesssim2$\,AU), with no evidence for a radio jet or extended emission. The compact radio source has an average flux density of $\sim0.37$\,mJy, and its brightness temperature is given at $\gtrsim2.3\times10^{7}$\,K, confirming a non-thermal origin. The emission exhibits irregular variations on a time-scale of tens of minutes, similar to the radio flares seen in AE Aqr. The measured VLBI position of J0240+1952 is consistent with that derived from \textit{Gaia}. Our results favour the model in which the radio emission is attributed to a superposition of synchrotron radiation from expanding magnetized blobs of this system.

astro-ph.SR↗
arXiv2023

Target-driven splitting SPH optimization of thermal conductivity distribution

Efficiently enhancing heat conduction through optimized distribution of a limited quantity of high thermal conductivity material is paramount in cooling electronic devices and numerous other applications. This paper introduces a target-driven all-at-once approach for PDE-constrained optimization and derives a splitting smoothed particle hydrodynamics (SPH) method for optimizing the distribution of thermal conductivity in heat conduction problems. In this method, the optimization iteration of the system is split into several easily addressed steps. A targeting step is employed to progressively enforce the direct target, which potentially leads to increased PDE residuals. Then, these residuals are recovered through an evolution step of the design variable. After this, a PDE solution step is carried out to further decrease the PDE residuals, and the system is ready for the next iteration. Unlike the simulation-based approaches, the present method does not rely on the adjoint state equation and converged state variable field in each iteration, and the optimization process is significantly simplified and accelerated. With the utilization of an implicit SPH splitting operator and a general numerical regularization formulation, the information propagation is further accelerated and the numerical stability is greatly enhanced. Typical examples of heat conduction optimization demonstrate that the current method yields optimal results comparable to previous methods and exhibits considerable computational efficiency. Moreover, the optimal results feature more moderate extreme values, which offers distinct advantages for the easier selection of appropriate material with high thermal conductivity.

cs.CE↗
arXiv2023

Transient dynamics of the phase transition in VO2 revealed by mega electron-volt ultrafast electron diffraction

Vanadium dioxide (VO2) exhibits an insulator-to-metal transition accompanied by a structural transition near room temperature. This transition can be triggered by an ultrafast laser pulse. Exotic transient states, such as a metallic state without structural transition, were also proposed. These unique characteristics let VO2 have great potential in thermal switchable devices and photonic applications. Although great efforts have been made, the atomic pathway during the photoinduced phase transition is still not clear. Here, we synthesized freestanding quasi-single-crystal VO2 films and examined their photoinduced structural phase transition with mega-electron-volt ultrafast electron diffraction. Leveraging the high signal-to-noise ratio and high temporal resolution, we observe that the disappearance of vanadium dimers and zigzag chains does not coincide with the transformation of crystal symmetry. After photoexcitation, the initial structure is strongly modified within 200 femtoseconds, resulting in a transient monoclinic structure without vanadium dimers and zigzag chains. Then, it continues to evolve to the final tetragonal structure in approximately 5 picoseconds. In addition, only one laser fluence threshold instead of two thresholds suggested in polycrystalline samples was observed in our quasi-single-crystal samples. Our findings provide new essential information for a comprehensive understanding of the photoinduced ultrafast phase transition in VO2.

cond-mat.mtrl-sci↗
arXiv2023

An improved method to measure $\rm ^{12}C/^{13}C$ and $\rm ^{14}N/^{15}N$ abundance ratios: revisiting CN isotopologues in the Galactic outer disk

The variations of elemental abundance and their ratios along the Galactocentric radius result from the chemical evolution of the Milky Way disks. The $\rm ^{12}C/^{13}C$ ratio in particular is often used as a proxy to determine other isotopic ratios, such as $\rm ^{16}O/^{18}O$ and $\rm ^{14}N/^{15}N$. Measurements of $\rm ^{12}CN$ and $\rm ^{13}CN$ (or $\rm C^{15}N$) -- with their optical depths corrected via their hyper-fine structure lines -- have traditionally been exploited to constrain the Galactocentric gradients of the CNO isotopic ratios. Such methods typically make several simplifying assumptions (e.g. a filling factor of unity, the Rayleigh-Jeans approximation, and the neglect of the cosmic microwave background) while adopting a single average gas phase. However, these simplifications introduce significant biases to the measured $\rm ^{12}C/^{13}C$ and $\rm ^{14}N/^{15}N$. We demonstrate that exploiting the optically thin satellite lines of $\rm ^{12}CN$ constitutes a more reliable new method to derive $\rm ^{12}C/^{13}C$ and $\rm ^{14}N/^{15}N$ from CN isotopologues. We apply this satellite-line method to new IRAM 30-m observations of $\rm ^{12}CN$, $\rm ^{13}CN$, and $\rm C^{15}N$ $N=1\to0$ towards 15 metal-poor molecular clouds in the Galactic outer disk ($R_{\rm gc} > $ 12 kpc), supplemented by data from the literature. After updating their Galactocentric distances, we find that $\rm ^{12}C/^{13}C$ and $\rm ^{14}N/^{15}N$ gradients are in good agreement with those derived using independent optically thin molecular tracers, even in regions with the lowest metallicities. We therefore recommend using optically thin tracers for Galactic and extragalactic CNO isotopic measurements, which avoids the biases associated with the traditional method.

astro-ph.GA↗
arXiv2023

A Speed Odyssey for Deployable Quantization of LLMs

The large language model era urges faster and less costly inference. Prior model compression works on LLMs tend to undertake a software-centric approach primarily focused on the simulated quantization performance. By neglecting the feasibility of deployment, these approaches are typically disabled in real practice. They used to drastically push down the quantization bit range for a reduced computation which might not be supported by the mainstream hardware, or involve sophisticated algorithms that introduce extra computation or memory access overhead. We argue that pursuing a hardware-centric approach in the construction of quantization algorithms is crucial. In this regard, we are driven to build our compression method on top of hardware awareness, eliminating impractical algorithm choices while maximizing the benefit of hardware acceleration. Our method, OdysseyLLM, comes with a novel W4A8 kernel implementation called FastGEMM and a combined recipe of quantization strategies. Extensive experiments manifest the superiority of our W4A8 method which brings the actual speed boosting up to \textbf{4$\times$} compared to Hugging Face FP16 inference and \textbf{2.23$\times$} vs. the state-of-the-art inference engine TensorRT-LLM in FP16, and \textbf{1.45$\times$} vs. TensorRT-LLM in INT8, yet without substantially harming the performance.

cs.LG↗
arXiv2023

Direct numerical simulation of compressible interfacial multiphase flows using a mass-momentum-energy consistent volume-of-fluid method

Compressible interfacial multiphase flows (CIMF) are essential to different applications, such as liquid fuel injection in supersonic propulsion systems. Since high-level details in CIMF are often difficult to measure in experiments, numerical simulation is an important alternative to shed light on the unclear physics. A direct numerical simulation (DNS) of CIMF will need to rigorously resolve the shock waves, the interfaces, and the interaction between the two. A novel numerical method has been developed and implemented in the present study. The geometric volume-of-fluid (VOF) method is employed to resolve the sharp interfaces between the two phases. The advection of the density, momentum, and energy is carried out consistently with VOF advection. To suppress spurious oscillations near shocks, numerical diffusion is introduced based on the Kurganov-Tadmor method in the region away from the interface. The contribution of pressure is incorporated using the projection method and the pressure is obtained by solving the Poisson-Helmholtz equation, which allows the present method to handle flows with all Mach numbers. The present method is tested by a sequence of CIMF problems. The simulation results are validated against theories, experiments, and other simulations, and excellent agreement has been achieved. In particular, the linear single-mode Richtmyer-Meshkov instabilities with finite Weber and Reynolds numbers are simulated. The simulation results agree very well with the linear stability theory, which affirms the capability of the present method in capturing the viscous and capillary effects on shock-interface interaction.

physics.flu-dyn↗
arXiv2023

MultiSPANS: A Multi-range Spatial-Temporal Transformer Network for Traffic Forecast via Structural Entropy Optimization

Traffic forecasting is a complex multivariate time-series regression task of paramount importance for traffic management and planning. However, existing approaches often struggle to model complex multi-range dependencies using local spatiotemporal features and road network hierarchical knowledge. To address this, we propose MultiSPANS. First, considering that an individual recording point cannot reflect critical spatiotemporal local patterns, we design multi-filter convolution modules for generating informative ST-token embeddings to facilitate attention computation. Then, based on ST-token and spatial-temporal position encoding, we employ the Transformers to capture long-range temporal and spatial dependencies. Furthermore, we introduce structural entropy theory to optimize the spatial attention mechanism. Specifically, The structural entropy minimization algorithm is used to generate optimal road network hierarchies, i.e., encoding trees. Based on this, we propose a relative structural entropy-based position encoding and a multi-head attention masking scheme based on multi-layer encoding trees. Extensive experiments demonstrate the superiority of the presented framework over several state-of-the-art methods in real-world traffic datasets, and the longer historical windows are effectively utilized. The code is available at https://github.com/SELGroup/MultiSPANS.

cs.LG↗
arXiv2023

Path-dependent correlations in dynamically tuned Ising models and its short-time behavior: application of Magnus expansion

We study the buildup of antiferromagnetic (AF) correlation in the dynamically tuned Ising models which are realized by the Rydberg atomic system. In short-time scale, we apply Magnus expansion (ME) to derive the high-order analytic expression of the connected correlation functions and compare it with exactly numerical results for the different lattice geometries, e.g., 1D chain, $2 \times n$ lattice, and $n \times n$ lattice. It is shown that the high-order expansion is required to describe accurately the buildup of AF correlation in the quench dynamics. Moreover, through a 2D square lattice, we find that the magnitude of AF correlation for the same Manhattan distance is proportional to the number of the shortest paths in a sufficiently long time until long and distinct paths are involved significantly with the buildup of the correlation. Finally, we propose an applicable experimental setup to realize our findings.

cond-mat.quant-gas↗
arXiv2023

On regular 2-path Hamiltonian graphs

Kronk introduced the $l$-path hamiltonianicity of graphs in 1969. A graph is $l$-path Hamiltonian if every path of length not exceeding $l$ is contained in a Hamiltonian cycle. We have shown that if $P=uvz$ is a 2-path of a 2-connected, $k$-regular graph on at most $2k$ vertices and $G - V(P)$ is connected, then there must exist a Hamiltonian cycle in $G$ that contains the 2-path $P$. In this paper, we characterize a class of graphs that illustrate the sharpness of the bound $2k$. Additionally, we show that by excluding the class of graphs, both 2-connected, $k$-regular graphs on at most $2k + 1$ vertices and 3-connected, $k$-regular graphs on at most $3k-6$ vertices satisfy that there is a Hamiltonian cycle containing the 2-path $P$ if $G\setminus V(P)$ is connected.

math.CO↗
arXiv2023

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations

Emotion recognition in conversations (ERC), the task of recognizing the emotion of each utterance in a conversation, is crucial for building empathetic machines. Existing studies focus mainly on capturing context- and speaker-sensitive dependencies on the textual modality but ignore the significance of multimodal information. Different from emotion recognition in textual conversations, capturing intra- and inter-modal interactions between utterances, learning weights between different modalities, and enhancing modal representations play important roles in multimodal ERC. In this paper, we propose a transformer-based model with self-distillation (SDT) for the task. The transformer-based model captures intra- and inter-modal interactions by utilizing intra- and inter-modal transformers, and learns weights between modalities dynamically by designing a hierarchical gated fusion strategy. Furthermore, to learn more expressive modal representations, we treat soft labels of the proposed model as extra training supervision. Specifically, we introduce self-distillation to transfer knowledge of hard and soft labels from the proposed model to each modality. Experiments on IEMOCAP and MELD datasets demonstrate that SDT outperforms previous state-of-the-art baselines.

cs.AI↗
arXiv2023

AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset

It is a long-term vision for Autonomous Driving (AD) community that the perception models can learn from a large-scale point cloud dataset, to obtain unified representations that can achieve promising results on different tasks or benchmarks. Previous works mainly focus on the self-supervised pre-training pipeline, meaning that they perform the pre-training and fine-tuning on the same benchmark, which is difficult to attain the performance scalability and cross-dataset application for the pre-training checkpoint. In this paper, for the first time, we are committed to building a large-scale pre-training point-cloud dataset with diverse data distribution, and meanwhile learning generalizable representations from such a diverse pre-training dataset. We formulate the point-cloud pre-training task as a semi-supervised problem, which leverages the few-shot labeled and massive unlabeled point-cloud data to generate the unified backbone representations that can be directly applied to many baseline models and benchmarks, decoupling the AD-related pre-training process and downstream fine-tuning task. During the period of backbone pre-training, by enhancing the scene- and instance-level distribution diversity and exploiting the backbone's ability to learn from unknown instances, we achieve significant performance gains on a series of downstream perception benchmarks including Waymo, nuScenes, and KITTI, under different baseline models like PV-RCNN++, SECOND, CenterPoint.

cs.CV↗
arXiv2023

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior

We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation. Code available at https://github.com/deepseek-ai/DreamCraft3D.

cs.CV↗
arXiv2023

Virtual Accessory Try-On via Keypoint Hallucination

The virtual try-on task refers to fitting the clothes from one image onto another portrait image. In this paper, we focus on virtual accessory try-on, which fits accessory (e.g., glasses, ties) onto a face or portrait image. Unlike clothing try-on, which relies on human silhouette as guidance, accessory try-on warps the accessory into an appropriate location and shape to generate a plausible composite image. In contrast to previous try-on methods that treat foreground (i.e., accessories) and background (i.e., human faces or bodies) equally, we propose a background-oriented network to utilize the prior knowledge of human bodies and accessories. Specifically, our approach learns the human body priors and hallucinates the target locations of specified foreground keypoints in the background. Then our approach will inject foreground information with accessory priors into the background UNet. Based on the hallucinated target locations, the warping parameters are calculated to warp the foreground. Moreover, this background-oriented network can also easily incorporate auxiliary human face/body semantic segmentation supervision to further boost performance. Experiments conducted on STRAT dataset validate the effectiveness of our proposed method.

cs.CV↗
arXiv2023

Multiple Key-value Strategy in Recommendation Systems Incorporating Large Language Model

Recommendation system (RS) plays significant roles in matching users information needs for Internet applications, and it usually utilizes the vanilla neural network as the backbone to handle embedding details. Recently, the large language model (LLM) has exhibited emergent abilities and achieved great breakthroughs both in the CV and NLP communities. Thus, it is logical to incorporate RS with LLM better, which has become an emerging research direction. Although some existing works have made their contributions to this issue, they mainly consider the single key situation (e.g. historical interactions), especially in sequential recommendation. The situation of multiple key-value data is simply neglected. This significant scenario is mainstream in real practical applications, where the information of users (e.g. age, occupation, etc) and items (e.g. title, category, etc) has more than one key. Therefore, we aim to implement sequential recommendations based on multiple key-value data by incorporating RS with LLM. In particular, we instruct tuning a prevalent open-source LLM (Llama 7B) in order to inject domain knowledge of RS into the pre-trained LLM. Since we adopt multiple key-value strategies, LLM is hard to learn well among these keys. Thus the general and innovative shuffle and mask strategies, as an innovative manner of data argument, are designed. To demonstrate the effectiveness of our approach, extensive experiments are conducted on the popular and suitable dataset MovieLens which contains multiple keys-value. The experimental results demonstrate that our approach can nicely and effectively complete this challenging issue.

cs.IR↗