arXiv subjects

Hao Tian

Publications and source records attributed to Hao Tian.

At least 73 records · Page 4Linked to original sources

arXiv2024

Tensor-involved peridynamics: A unified framework for isotropic and anisotropic materials

In this paper, we introduce tensor involved peridynamics, a unified framework for simulating both isotropic and anisotropic materials. While traditional peridynamics models effectively simulate isotropic materials, they face challenges with anisotropic materials and are prone to instability caused by zero energy modes. Our novel model extend the linear bond based peridynamics framework by incorporating the elastic tensor into the micrmodulus function, thereby ensuring stability for anisotropic materials without the need for additional corrections. For isotropic materials. the model mantains compatibility with conventional bond based peridynamics, assuming Possion's rations of 1/4 in 3D and 1/3 in 2D.Numerical experiments confirm the model's stability and accuracy across various scenarios. Additionally, we introduce a damage model for isotropic materials. validating its performance in predicting crack propagation paths in a 2D plate. The results show superior alignment with experimental date compared to traditional model.

cond-mat.mtrl-sci↗

arXiv2024

PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models

Large Vision-Language Models (VLMs) have been extended to understand both images and videos. Visual token compression is leveraged to reduce the considerable token length of visual inputs. To meet the needs of different tasks, existing high-performance models usually process images and videos separately with different token compression strategies, limiting the capabilities of combining images and videos. To this end, we extend each image into a "static" video and introduce a unified token compression strategy called Progressive Visual Token Compression (PVC), where the tokens of each frame are progressively encoded and adaptively compressed to supplement the information not extracted from previous frames. Video tokens are efficiently compressed with exploiting the inherent temporal redundancy. Images are repeated as static videos, and the spatial details can be gradually supplemented in multiple frames. PVC unifies the token compressing of images and videos. With a limited number of tokens per frame (64 tokens by default), spatial details and temporal changes can still be preserved. Experiments show that our model achieves state-of-the-art performance across various video understanding benchmarks, including long video tasks and fine-grained short video tasks. Meanwhile, our unified token compression strategy incurs no performance loss on image benchmarks, particularly in detail-sensitive tasks.

cs.CV↗

arXiv2024

A mechanical quantum memory for microwave photons

Long-lived mechanical oscillators are actively pursued as critical resources for quantum storage, sensing, and transduction. However, achieving deterministic quantum control while limiting mechanical dissipation remains a persistent challenge. Here, we demonstrate strong coupling between a transmon superconducting qubit and an ultra-long-lived nanomechanical oscillator ($T_\text{1} \approx 25 \text{ ms}$ at 5 GHz, $Q \approx 0.8 \times 10^9$) by leveraging the low acoustic loss in silicon and phononic bandgap engineering. The qubit-oscillator system achieves large cooperativity ($C_{T_1}\approx 1.5\times10^5$, $C_{T_2}\approx 150$), enabling the generation of non-classical states and the investigation of mechanisms underlying mechanical decoherence. We show that dynamical decoupling$\unicode{x2014}$implemented through the qubit$\unicode{x2014}$can mitigate decoherence, leading to a mechanical coherence time of $T_2\approx 1 \text{ ms}$. These findings extend the exceptional storage capabilities of mechanical oscillators to the quantum regime, putting them forward as compact bosonic elements for future applications in quantum computing and metrology.

quant-ph↗

arXiv2024

Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation

Large language models (LLMs) have shown impressive prowess in solving a wide range of tasks with world knowledge. However, it remains unclear how well LLMs are able to perceive their factual knowledge boundaries, particularly under retrieval augmentation settings. In this study, we present the first analysis on the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain question answering (QA), with a bunch of important findings. Specifically, we focus on three research questions and analyze them by examining QA, priori judgement and posteriori judgement capabilities of LLMs. We show evidence that LLMs possess unwavering confidence in their knowledge and cannot handle the conflict between internal and external knowledge well. Furthermore, retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We further conduct thorough experiments to examine how different factors affect LLMs and propose a simple method to dynamically utilize supporting documents with our judgement strategy. Additionally, we find that the relevance between the supporting documents and the questions significantly impacts LLMs' QA and judgemental capabilities. The code to reproduce this work is available at https://github.com/RUCAIBox/LLM-Knowledge-Boundary.

cs.CL↗

arXiv2024

Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance

Multimodal large language models (MLLMs) have demonstrated impressive performance in vision-language tasks across a broad spectrum of domains. However, the large model scale and associated high computational costs pose significant challenges for training and deploying MLLMs on consumer-grade GPUs or edge devices, thereby hindering their widespread application. In this work, we introduce Mini-InternVL, a series of MLLMs with parameters ranging from 1B to 4B, which achieves 90% of the performance with only 5% of the parameters. This significant improvement in efficiency and effectiveness makes our models more accessible and applicable in various real-world scenarios. To further promote the adoption of our models, we develop a unified adaptation framework for Mini-InternVL, which enables our models to transfer and outperform specialized models in downstream tasks, including autonomous driving, medical images, and remote sensing. We believe that our study can provide valuable insights and resources to advance the development of efficient and effective MLLMs. Code is available at https://github.com/OpenGVLab/InternVL.

cs.CV↗

arXiv2024

Assessing the Association between the Globular Cluster NGC 4147 and the Sagittarius Dwarf Galaxy

The potential association of the globular cluster (GC) NGC 4147 with the Sagittarius (Sgr) dwarf spheroidal galaxy has been proposed due to their comparable locations and radial velocities. However, there are still debates about this connection. In this study, we use data from the Dark Energy Spectroscopic Instrument Legacy Imaging Surveys to assess their association. We redetermine thefundamental parameters of NGC 4147 and find that the cluster is 11.0 Gyr old, has a metallicity of Z=0.0006, and is located 18.5 kpc from the Sun. We utilize the matched filter algorithm to identify extratidal structures in the surrounding sky of NGC 4147. The multiarmed tidal structures we find align more closely with the result of internal two-body relaxation processes within the cluster itself. The orientations of the dispersed tidal structures, the orbital direction of the cluster, and the mean orbital direction of Sgr do not show any apparent connection to each other. It seems to challenge the hypothesis of a common origin between the cluster and Sgr. To further investigate the association, we study the kinematics of NGC 4147 with the newly determined fundamental parameters. We find that the orbit, orbital energy, and angular momentum of NGC 4147 are not compatible with those of Sgr or its streams. This suggests that the cluster is not dynamically associated with Sgr. The morphology and dynamics of NGC 4147 are more consistent with it being a GC that formed with other origin rather than being accreted from the Sgr dwarf galaxy.

astro-ph.GA↗

arXiv2024

RetriBooru: Leakage-Free Retrieval of Conditions from Reference Images for Subject-Driven Generation

Diffusion-based methods have demonstrated remarkable capabilities in generating a diverse array of high-quality images, sparking interests for styled avatars, virtual try-on, and more. Previous methods use the same reference image as the target. An overlooked aspect is the leakage of the target's spatial information, style, etc. from the reference, harming the generated diversity and causing shortcuts. However, this approach continues as widely available datasets usually consist of single images not grouped by identities, and it is expensive to recollect large-scale same-identity data. Moreover, existing metrics adopt decoupled evaluation on text alignment and identity preservation, which fail at distinguishing between balanced outputs and those that over-fit to one aspect. In this paper, we propose a multi-level, same-identity dataset RetriBooru, which groups anime characters by both face and cloth identities. RetriBooru enables adopting reference images of the same character and outfits as the target, while keeping flexible gestures and actions. We benchmark previous methods on our dataset, and demonstrate the effectiveness of training with a reference image different from target (but same identity). We introduce a new concept composition task, where the conditioning encoder learns to retrieve different concepts from several reference images, and modify a baseline network RetriNet for the new task. Finally, we introduce a novel class of metrics named Similarity Weighted Diversity (SWD), to measure the overlooked diversity and better evaluate the alignment between similarity and diversity.

cs.CV↗

arXiv2024

PUMA: Empowering Unified MLLM with Multi-granular Visual Generation

Recent advancements in multimodal foundation models have yielded significant progress in vision-language understanding. Initial attempts have also explored the potential of multimodal large language models (MLLMs) for visual content generation. However, existing works have insufficiently addressed the varying granularity demands of different image generation tasks within a unified MLLM paradigm - from the diversity required in text-to-image generation to the precise controllability needed in image manipulation. In this work, we propose PUMA, emPowering Unified MLLM with Multi-grAnular visual generation. PUMA unifies multi-granular visual features as both inputs and outputs of MLLMs, elegantly addressing the different granularity requirements of various image generation tasks within a unified MLLM framework. Following multimodal pretraining and task-specific instruction tuning, PUMA demonstrates proficiency in a wide range of multimodal tasks. This work represents a significant step towards a truly unified MLLM capable of adapting to the granularity demands of various visual tasks. The code and model will be released in https://github.com/rongyaofang/PUMA.

cs.CV↗

arXiv2024

The Star Formation History of Nearby Galaxies: A Machine Learning Approach

Reproducing color-magnitude diagrams (CMDs) of star-resolved galaxies is one of the most precise methods for measuring the star formation history (SFH) of nearby galaxies back to the earliest time. The upcoming big data era poses challenges to the traditional numerical technique in its capacity to deal with vast amounts of data, which motivates us to explore the feasibility of employing machine learning networks in this field. In this study, we refine the synthetic CMD method with a state-of-the-art theoretical stellar evolution model to simulate the properties of stellar populations, incorporate the convolutional neural network (CNN) in the fitting process to enhance the efficiency, and innovate the initial stellar mass estimation to improve the flexibility. The fine-tuned deep learning network, named \texttt{SFHNet}, has been tested with synthetic data and further validated with photometric data collected from the Hubble Space Telescope (\textit{HST}). The derived SFHs are largely in accordance with those reported in the literature. Furthermore, the network provides detailed insights into the distribution of stellar density, initial stellar mass, and star formation rate (SFR) over the age-metallicity map. The application of the deep learning network not only measures the SFH accurately but also enhances the synthetic CMD method's efficiency and flexibility, thereby facilitating a more comprehensive and in-depth understanding of nearby galaxies.

astro-ph.GA↗

arXiv2024

Uncovering the first-infall history of the LMC through its dynamical impact in the Milky Way halo

The gravitational interactions between the LMC and the Milky Way cause dynamical perturbations in the MW halo, leading to biased distributions of stellar density and kinematics. We run 50 high-resolution N-body simulations exploring varying masses and halo shapes of the MW and LMC to study the evolution of LMC-induced perturbations. By measuring mean velocities of simulated halo stars, we identify a discontinuity between the first-infall and second-passage scenarios of the LMC's orbital history. In the first infall, the Galactocentric latitudinal velocity hovers around 16 km/s for stars at 50-100 kpc, while it subsides to about 8 km/s in the second-passage scenario. We demonstrate that this reduced perturbation magnitude in the second-passage scenario is mainly due to the short dynamical times of the Galactic inner halo and the lower velocity of the LMC during its second infall into the MW. Using a subset of $\sim 1100$ RR Lyrae stars located in the outer halo ($50 \leq R_{\mathrm{GC}} < 100$ kpc) with precise distance estimates from Gaia, we find the mean latitudinal velocity ($v_{b}$) in the Galactocentric frame to be $\langle v_{b} \rangle = 18.1 \pm 4.1$ km/s. The observation supports the first-infall scenario with a massive LMC ($\sim 2.1 \times 10^{11} \mathrm{M}_{\odot}$) at infall, an oblate MW halo with a virial mass $M_{200} < 1.4 \times 10^{12} \mathrm{M}_{\odot}$ and a flattening parameter $q > 0.7$. Our study indicates that LMC-induced kinematic disturbances can reveal its orbital history and key characteristics, as well as those of the MW. This approach shows promise in helping determine fundamental parameters of both galaxies.

astro-ph.GA↗

arXiv2024

The Velocity Aberration Effect of the CSST Main Survey Camera

In this study, we conducted simulations to find the geometric aberrations expected for images taken by the Main Survey Camera (MSC) of the Chinese Space Station Telescope (CSST) due to its motion. As anticipated by previous work, our findings indicate that the geometric distortion of light impacts the focal plane's apparent scale, with a more pronounced influence as the size of the focal plane increases. Our models suggest that the effect consistently influences the pixel scale in both the vertical and parallel directions. The apparent scale variation follows a sinusoidal distribution throughout one orbit period. Simulations reveal that the effect is particularly pronounced in the center of the Galaxy and gradually diminishes along the direction of ecliptic latitude. At low ecliptic latitudes, the total aberration leads to about 0.94 pixels offset (a 20-minute exposure) and 0.26 pixels offset (a 300-second exposure) at the edge of the field of view, respectively. Appropriate processings for the geometric effect during the CSST pre- and post-observation phases are presented.

astro-ph.IM↗

arXiv2024

MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity

Despite the effectiveness of vision-language supervised fine-tuning in enhancing the performance of Vision Large Language Models (VLLMs). However, existing visual instruction tuning datasets include the following limitations: (1) Instruction annotation quality: despite existing VLLMs exhibiting strong performance, instructions generated by those advanced VLLMs may still suffer from inaccuracies, such as hallucinations. (2) Instructions and image diversity: the limited range of instruction types and the lack of diversity in image data may impact the model's ability to generate diversified and closer to real-world scenarios outputs. To address these challenges, we construct a high-quality, diverse visual instruction tuning dataset MMInstruct, which consists of 973K instructions from 24 domains. There are four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering and Short Visual Question Answering. To construct MMInstruct, we propose an instruction generation data engine that leverages GPT-4V, GPT-3.5, and manual correction. Our instruction generation engine enables semi-automatic, low-cost, and multi-domain instruction generation at 1/6 the cost of manual construction. Through extensive experiment validation and ablation experiments, we demonstrate that MMInstruct could significantly improve the performance of VLLMs, e.g., the model fine-tuning on MMInstruct achieves new state-of-the-art performance on 10 out of 12 benchmarks. The code and data shall be available at https://github.com/yuecao0119/MMInstruct.

cs.CV↗

arXiv2024

MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

The capability to process multiple images is crucial for Large Vision-Language Models (LVLMs) to develop a more thorough and nuanced understanding of a scene. Recent multi-image LVLMs have begun to address this need. However, their evaluation has not kept pace with their development. To fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess LVLMs across a wide range of multi-image tasks. MMIU encompasses 7 types of multi-image relationships, 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions, making it the most extensive benchmark of its kind. Our evaluation of 24 popular LVLMs, including both open-source and proprietary models, reveals significant challenges in multi-image comprehension, particularly in tasks involving spatial understanding. Even the most advanced models, such as GPT-4o, achieve only 55.7% accuracy on MMIU. Through multi-faceted analytical experiments, we identify key performance gaps and limitations, providing valuable insights for future model and data improvements. We aim for MMIU to advance the frontier of LVLM research and development, moving us toward achieving sophisticated multimodal multi-image user interactions.

cs.CV↗

arXiv2024

Piezoelectric actuation for integrated photonics

Recent decades have seen significant advancements in integrated photonics, driven by improvements in nanofabrication technology. This field has developed from integrated semiconductor lasers and low-loss waveguides to optical modulators, enabling the creation of sophisticated optical systems on a chip scale capable of performing complex functions like optical sensing, signal processing, and metrology. The tight confinement of optical modes in photonic waveguides further enhances the optical nonlinearity, leading to a variety of nonlinear optical phenomena such as optical frequency combs, second-harmonic generation, and supercontinuum generation. Active tuning of photonic circuits is crucial not only for offsetting variations caused by fabrication in large-scale integration, but also serves as a fundamental component in programmable photonic circuits. Piezoelectric actuation in photonic devices offers a low-power, high-speed solution and is essential in the design of future photonic circuits due to its compatibility with materials like Si and Si3N4, which do not exhibit electro-optic effects. Here, we provide a detailed review of the latest developments in piezoelectric tuning and modulation, by examining various piezoelectric materials, actuator designs tailored to specific applications, and the capabilities and limitations of current technologies. Additionally, we explore the extensive applications enabled by piezoelectric actuators, including tunable lasers, frequency combs, quantum transducers, and optical isolators. These innovative ways of managing photon propagation and frequency on-chip are expected to be highly sought after in the future advancements of advanced photonic chips for both classical and quantum optical information processing and computing.

physics.optics↗

arXiv2024

OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text

Image-text interleaved data, consisting of multiple images and texts arranged in a natural document format, aligns with the presentation paradigm of internet data and closely resembles human reading habits. Recent studies have shown that such data aids multimodal in-context learning and maintains the capabilities of large language models during multimodal fine-tuning. However, the limited scale and diversity of current image-text interleaved data restrict the development of multimodal large language models. In this paper, we introduce OmniCorpus, a 10 billion-scale image-text interleaved dataset. Using an efficient data engine, we filter and extract large-scale high-quality documents, which contain 8.6 billion images and 1,696 billion text tokens. Compared to counterparts (e.g., MMC4, OBELICS), our dataset 1) has 15 times larger scales while maintaining good data quality; 2) features more diverse sources, including both English and non-English websites as well as video-centric websites; 3) is more flexible, easily degradable from an image-text interleaved format to pure text corpus and image-text pairs. Through comprehensive analysis and experiments, we validate the quality, usability, and effectiveness of the proposed dataset. We hope this could provide a solid data foundation for future multimodal model research. Code and data are released at https://github.com/OpenGVLab/OmniCorpus.

cs.CV↗

arXiv2024

Detailed Mapping of the Galactic Disk Structure in the Solar Neighborhood through LAMOST K Dwarfs

The Galactic disk is one of the main components of the Milky Way, which contributes most of the luminosity. Its structure is essential for understanding the formation and evolution of the Milky Way. Using 174,443 K-type dwarf stars observed by both LAMOST and Gaia DR3, we study the disk density profile in the local volume within 1,200 pc. In the azimuthal dimension, we find strong asymmetric signal of the thin disk. The surface density and the scale height of the southern disk significantly change versus the azimuthal angle at the same galactocentric distance $R$. Meanwhile, in the vertical dimension, the scale height of the northern disk has quite different trend than that of the southern one. The scale height of the southern disk shows a decreasing trend with $ϕ\sim-2.5^\circ$, and change to an increasing one with $ϕ\sim5.0^°$. Meanwhile, the scale height of the northern disk has a consistently smaller increase. Finally, we divide the entire sample into three subsamples based on metallicity and all three subsamples show significant non-axisymmetric and north-south asymmetric signals in the Galactic disk. Furthermore, we find that the scale height of the metal-poor ([Fe/H] $<$ -0.4 dex) subsample in the northern disk is greater than that of the metal-rich ([Fe/H] $>$ -0.1 dex) subsample. However, in the southern disk, the scale height exhibits varying relationships across different metallicity slices.

astro-ph.GA↗

arXiv2024

JumpCoder: Go Beyond Autoregressive Coder via Online Modification

While existing code large language models (code LLMs) exhibit impressive capabilities in code generation, their autoregressive sequential generation inherently lacks reversibility. This limitation hinders them from timely correcting previous missing statements during coding as humans do, often leading to error propagation and suboptimal performance. We introduce JumpCoder, a novel model-agnostic framework that enables human-like online modification and non-sequential generation to augment code LLMs. The key idea behind JumpCoder is to insert new code into the currently generated code when necessary during generation, which is achieved through an auxiliary infilling model that works in tandem with the code LLM. Since identifying the best infill position beforehand is intractable, we adopt an \textit{infill-first, judge-later} strategy, which experiments with filling at the $k$ most critical positions following the generation of each line, and uses an Abstract Syntax Tree (AST) parser alongside the Generation Model Scoring to effectively judge the validity of each potential infill. Extensive experiments using six state-of-the-art code LLMs across multiple and multilingual benchmarks consistently indicate significant improvements over all baselines. Our code is public at https://github.com/Keytoyze/JumpCoder.

cs.CL↗

arXiv2024

Brain Imaging-to-Graph Generation using Adversarial Hierarchical Diffusion Models for MCI Causality Analysis

Effective connectivity can describe the causal patterns among brain regions. These patterns have the potential to reveal the pathological mechanism and promote early diagnosis and effective drug development for cognitive disease. However, the current methods utilize software toolkits to extract empirical features from brain imaging to estimate effective connectivity. These methods heavily rely on manual parameter settings and may result in large errors during effective connectivity estimation. In this paper, a novel brain imaging-to-graph generation (BIGG) framework is proposed to map functional magnetic resonance imaging (fMRI) into effective connectivity for mild cognitive impairment (MCI) analysis. To be specific, the proposed BIGG framework is based on the diffusion denoising probabilistic models (DDPM), where each denoising step is modeled as a generative adversarial network (GAN) to progressively translate the noise and conditional fMRI to effective connectivity. The hierarchical transformers in the generator are designed to estimate the noise at multiple scales. Each scale concentrates on both spatial and temporal information between brain regions, enabling good quality in noise removal and better inference of causal relations. Meanwhile, the transformer-based discriminator constrains the generator to further capture global and local patterns for improving high-quality and diversity generation. By introducing the diffusive factor, the denoising inference with a large sampling step size is more efficient and can maintain high-quality results for effective connectivity generation. Evaluations of the ADNI dataset demonstrate the feasibility and efficacy of the proposed model. The proposed model not only achieves superior prediction performance compared with other competing methods but also predicts MCI-related causal connections that are consistent with clinical studies.

cs.CV↗