arXiv subjects

Yu Liu

Publications and source records attributed to Yu Liu.

At least 505 records · Page 28Linked to original sources

arXiv2024

The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

We present The Matrix, the first foundational realistic world simulator capable of generating continuous 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives, enabling immersive exploration of richly dynamic environments. Trained on limited supervised data from AAA games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains -- deserts, grasslands, water bodies, and urban landscapes -- in continuous, uncut hour-long sequences. Operating at 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting--an environment present in neither gaming data nor real-world sources. This approach showcases the potential of AAA game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data.

cs.AI↗

arXiv2024

Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective

Generative models, particularly diffusion models, have achieved remarkable success in density estimation for multimodal data, drawing significant interest from the reinforcement learning (RL) community, especially in policy modeling in continuous action spaces. However, existing works exhibit significant variations in training schemes and RL optimization objectives, and some methods are only applicable to diffusion models. In this study, we compare and analyze various generative policy training and deployment techniques, identifying and validating effective designs for generative policy algorithms. Specifically, we revisit existing training objectives and classify them into two categories, each linked to a simpler approach. The first approach, Generative Model Policy Optimization (GMPO), employs a native advantage-weighted regression formulation as the training objective, which is significantly simpler than previous methods. The second approach, Generative Model Policy Gradient (GMPG), offers a numerically stable implementation of the native policy gradient method. We introduce a standardized experimental framework named GenerativeRL. Our experiments demonstrate that the proposed methods achieve state-of-the-art performance on various offline-RL datasets, offering a unified and practical guideline for training and deploying generative policies.

cs.LG↗

arXiv2024

Geographic Space as Manifolds

The communications and interrelations between different locations on the Earth's surface have far-reaching implications for both social and natural systems. Effective spatial analytics ideally require a spatial representation, where geographic principles are succinctly expressed within a defined metric space. However, common spatial representations, including map-based or network-based approaches, fall short by incompletely or inaccurately defining this metric space. Here we show, by introducing an inverse friction factor that captures the spatial constraints in spatial networks, that a homogeneous, low-dimensional spatial representation - termed the Geographic Manifold - can be achieved. We illustrate the effectiveness of the Geographic Manifold in two classic scenarios of spatial analytics - location choice and propagation, where the otherwise complicated analyses are reduced to straightforward regular partitioning and concentric diffusing, respectively on the manifold with a high degree of accuracy. We further empirically explain and formally prove the general existence of the Geographic Manifold, which is grounded in the intrinsic Euclidean low-dimensional statistical physics properties of geographic phenomena. This work represents a step towards formalizing Tobler's famous First Law of Geography from a geometric approach, where a regularized geospace thereby yielded is expected to contribute in learning abstract spatial structure representations for understanding and optimization purposes.

physics.soc-ph↗

arXiv2024

CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching

Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images is still challenging. The root reason behind the misalignment has not been extensively investigated. We observe that the misalignment is caused by inadequate token attention activation. We further attribute this phenomenon to the diffusion model's insufficient condition utilization, which is caused by its training paradigm. To address the issue, we propose CoMat, an end-to-end diffusion model fine-tuning strategy with an image-to-text concept matching mechanism. We leverage an image captioning model to measure image-to-text alignment and guide the diffusion model to revisit ignored tokens. A novel attribute concentration module is also proposed to address the attribute binding problem. Without any image or human preference data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL. Extensive experiments show that CoMat-SDXL significantly outperforms the baseline model SDXL in two text-to-image alignment benchmarks and achieves start-of-the-art performance.

cs.CV↗

arXiv2024

MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines

The advent of Large Language Models (LLMs) has paved the way for AI search engines, e.g., SearchGPT, showcasing a new paradigm in human-internet interaction. However, most current AI search engines are limited to text-only settings, neglecting the multimodal user queries and the text-image interleaved nature of website information. Recently, Large Multimodal Models (LMMs) have made impressive strides. Yet, whether they can function as AI search engines remains under-explored, leaving the potential of LMMs in multimodal search an open question. To this end, we first design a delicate pipeline, MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On top of this, we introduce MMSearch, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs. The curated dataset contains 300 manually collected instances spanning 14 subfields, which involves no overlap with the current LMMs' training data, ensuring the correct answer can only be obtained within searching. By using MMSearch-Engine, the LMMs are evaluated by performing three individual tasks (requery, rerank, and summarization), and one challenging end-to-end task with a complete searching process. We conduct extensive experiments on closed-source and open-source LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best results, which surpasses the commercial product, Perplexity Pro, in the end-to-end task, demonstrating the effectiveness of our proposed pipeline. We further present error analysis to unveil current LMMs still struggle to fully grasp the multimodal search tasks, and conduct ablation study to indicate the potential of scaling test-time computation for AI search engine. We hope MMSearch may provide unique insights to guide the future development of multimodal AI search engine. Project Page: https://mmsearch.github.io

cs.CV↗

arXiv2024

Investigating the Behavior and Spatiotemporal Variations of Green Line Emission in the Solar Corona

Understanding coronal structure and dynamics can be facilitated by analyzing green-line emission, which enables the investigation of diverse coronal structures such as coronal loops, streamers, coronal holes, and various eruptions in the solar atmosphere. In this study, we investigated the spatiotemporal behaviors of green-line emissions in both low and high latitudes across nine solar cycles, ranging from cycle 17 to the current cycle 25, using the Modified Homogeneous Data Set (MHDS). We employed methodologies such as cross-correlation, power spectral density (PSD), and wavelet transform techniques for this analysis. We found distinct behaviors in green line energy across various latitudinal distributions in the solar atmosphere. The trends observed at higher latitudes differ from those at lower latitudes. The emission behaviors show a close association with other solar phenomena like solar flares, sunspots, and coronal mass ejections (CMEs) throughout the solar cycles. The observed variations exhibit harmonic periods. The emission activity is significantly higher in the low latitudes, accounting for over 70 percent of the emissions, while the higher latitudes contribute less than 30 percent. The emissions exhibit asymmetric behavior between the northern and southern hemispheres, leading to a 44-year cycle of solar hemispheric dominance shifts. Various factors, such as Alfvén waves, solar magnetic fields, sunspots, differential rotation, and reconnection events, influence the observed differences in behavior between lower and higher latitudes, suggesting the existence of potential underlying phenomena contributing to deviations in properties, intensity, temporal dynamics, and spatiotemporal lifetime.

astro-ph.SR↗

arXiv2024

CPGA: Coding Priors-Guided Aggregation Network for Compressed Video Quality Enhancement

Recently, numerous approaches have achieved notable success in compressed video quality enhancement (VQE). However, these methods usually ignore the utilization of valuable coding priors inherently embedded in compressed videos, such as motion vectors and residual frames, which carry abundant temporal and spatial information. To remedy this problem, we propose the Coding Priors-Guided Aggregation (CPGA) network to utilize temporal and spatial information from coding priors. The CPGA mainly consists of an inter-frame temporal aggregation (ITA) module and a multi-scale non-local aggregation (MNA) module. Specifically, the ITA module aggregates temporal information from consecutive frames and coding priors, while the MNA module globally captures spatial information guided by residual frames. In addition, to facilitate research in VQE task, we newly construct the Video Coding Priors (VCP) dataset, comprising 300 videos with various coding priors extracted from corresponding bitstreams. It remedies the shortage of previous datasets on the lack of coding information. Experimental results demonstrate the superiority of our method compared to existing state-of-the-art methods. The code and dataset will be released at https://github.com/VQE-CPGA/CPGA.git .

eess.IV↗

arXiv2024

Phenomenological relationship between eccentric and quasi-circular orbital binary black hole waveform

Eccentricity, an important parameter of gravitational waves, has been paid more and more attention because it can reflect the dynamics of compact object mergers. Obtaining an accurate and fast gravitational waveform template is of great significance for the estimation of gravitational wave parameters. This paper aims to do an extended study of the phenomenological fitting model proposed by Setyawati and Ohme for adding eccentricity to quasi-circular orbital waveforms. It can be applied to higher eccentricity up to e = 0.4. But the higher the eccentricity, the less the accuracy. For e in [0, 0.1], it gives an overlap of more than 99.99%. For e in [0.1, 0.2], it gives an overlap of more than 99.9%. For e in [0.2, 0.3], it gives an overlap of more than 99%. For e in [0.3, 0.4], it gives an overlap of more than 90%. The reason for these phenomena is that the larger the eccentricity, the larger the deviation of the eccentricity estimator from the cosine function due to the large change in the morphology of the eccentric waveform, and the worse the fitting effect of the model. It can be applied to higher-order modes and gives the same overlap behavior. After adding a shift parameter, it can be applied to spin-aligned or spin-antialigned waveforms. After obtaining spin-precessing effect, it can be applied to the spin-precessing case. In summary, non-spining, spin-aligned, spin-antialigned or spin-precessing waveforms with eccentricity can be constructed from quasi-circular non-spining waveforms by the phenomenological model, which is not only helpful for us to quickly construct phenomenological gravitational wave templates, but also reveals a phenomenological and universal relationship between eccentric waveform and quasi-circular orbital waveform.

gr-qc↗

arXiv2024

Characterizing the effect of eccentricity on the dynamics of binary black hole mergers in numerical relativity

Many articles have partially studied the configuration of eccentric orbital binary black hole (BBH) mergers. However, there is a scarcity of systematic and comprehensive research on the effect of eccentricity on BBH dynamics. Thanks to the rich and numerous numerical relativistic simulations of eccentric orbital BBH mergers from RIT catalog, this paper aims to investigate the impact of initial eccentricity $e_0$ on various dynamic quantities such as merger time $T_{\text{merger}}$, peak luminosity $L_{\text{peak}}$ of gravitational waves, recoil velocity $V_f$, mass $M_f$, and spin $α_f$ of merger remnants. We cover configurations of no spin, spin alignment, and spin precession, as well as a broad parameter space of mass ratio ranging from 1/32 to 1 and initial eccentricity from 0 to 1. For non-spinning BBH with an initial coordinate separation of $11.3M$ ($M$ is the total mass of BBH), we make the first discovery of a ubiquitous oscillation in the relationship between dynamic quantities $L_{\text{peak}}$, $V_f$, $M_f$, $α_f$, and initial eccentricity $e_0$. Additionally, at $24.6M$, we observe the same oscillation phenomenon in the case of mass ratio $q=1$, but do not see it in other mass ratios, suggesting that this oscillation will be evident in numerical simulations with sufficiently dense initial eccentricity. abbreviated

gr-qc↗

arXiv2024

Reward driven workflows for unsupervised explainable analysis of phases and ferroic variants from atomically resolved imaging data

Rapid progress in aberration corrected electron microscopy necessitates development of robust methods for the identification of phases, ferroic variants, and other pertinent aspects of materials structure from imaging data. While unsupervised methods for clustering and classification are widely used for these tasks, their performance can be sensitive to hyperparameter selection in the analysis workflow. In this study, we explore the effects of descriptors and hyperparameters on the capability of unsupervised ML methods to distill local structural information, exemplified by discovery of polarization and lattice distortion in Sm doped BiFeO3 (BFO) thin films. We demonstrate that a reward-driven approach can be used to optimize these key hyperparameters across the full workflow, where rewards were designed to reflect domain wall continuity and straightness, ensuring that the analysis aligns with the material's physical behavior. This approach allows us to discover local descriptors that are best aligned with the specific physical behavior, providing insight into the fundamental physics of materials. We further extend the reward driven workflows to disentangle structural factors of variation via optimized variational autoencoder (VAE). Finally, the importance of well-defined rewards was explored as a quantifiable measure of success of the workflow.

cond-mat.mtrl-sci↗

arXiv2024

MTFusion: Reconstructing Any 3D Object from Single Image Using Multi-word Textual Inversion

Reconstructing 3D models from single-view images is a long-standing problem in computer vision. The latest advances for single-image 3D reconstruction extract a textual description from the input image and further utilize it to synthesize 3D models. However, existing methods focus on capturing a single key attribute of the image (e.g., object type, artistic style) and fail to consider the multi-perspective information required for accurate 3D reconstruction, such as object shape and material properties. Besides, the reliance on Neural Radiance Fields hinders their ability to reconstruct intricate surfaces and texture details. In this work, we propose MTFusion, which leverages both image data and textual descriptions for high-fidelity 3D reconstruction. Our approach consists of two stages. First, we adopt a novel multi-word textual inversion technique to extract a detailed text description capturing the image's characteristics. Then, we use this description and the image to generate a 3D model with FlexiCubes. Additionally, MTFusion enhances FlexiCubes by employing a special decoder network for Signed Distance Functions, leading to faster training and finer surface representation. Extensive evaluations demonstrate that our MTFusion surpasses existing image-to-3D methods on a wide range of synthetic and real-world images. Furthermore, the ablation study proves the effectiveness of our network designs.

cs.CV↗

arXiv2024

Reward based optimization of resonance-enhanced piezoresponse spectroscopy

Dynamic spectroscopies in Scanning Probe Microscopy (SPM) are critical for probing material properties, such as force interactions, mechanical properties, polarization switching, and electrochemical reactions and ionic dynamics. However, the practical implementation of these measurements is constrained by the need to balance imaging time and data quality. Signal to noise requirements favor long acquisition times and high frequencies to improve signal fidelity. However, these are limited on the low end by contact resonant frequency and photodiode sensitivity, and on the high end by the time needed to acquire high-resolution spectra, or the propensity for samples degradation under high field excitation over long times. The interdependence of key parameters such as instrument settings, acquisition times, and sampling rates makes manual tuning labor-intensive and highly dependent on user expertise, often yielding operator-dependent results. These limitations are prominent in techniques like Dual Amplitude Resonance Tracking (DART) in Piezoresponse Force Microscopy (PFM) that utilize multiple concurrent feedback loops for topography and resonance frequency tracking. Here, a reward-driven workflow is proposed that automates the tuning process, adapting experimental conditions in real time to optimize data quality. This approach significantly reduces the complexity and time required for manual adjustments and can be extended to other SPM spectroscopic methods, enhancing overall efficiency and reproducibility.

cond-mat.mtrl-sci↗

arXiv2024

Imaging half-unit-cell Cooper-pair density waves in monolayer 1T$^{\prime}$-MoTe$_2$

Unconventional superconductors that spontaneously break space-group symmetries of their underlying crystal lattice are distinguished by spatial modulations of the superconducting order parameter. These states have recently captured significant attention in various strongly correlated materials, where the breaking of translational or intra-unit-cell symmetries results in the emergence of pair density waves with wavelengths extending across one or multiple unit cells. Here, we employ a spectroscopic-imaging scanning tunneling microscope to identify robust half-unit-cell modulations of the superconducting gap magnitude, coherence strength and subgap states in tunable 1T$^{\prime}$-MoTe$_2$ monolayer. The modulations are oriented perpendicular to the zigzag Mo chains and coexist with three-unit-cell primary pair density wave modulations along the Mo chains, both of which attenuate with increasing temperature and the application of external magnetic fields. Importantly, we find that the superconductivity modulations are strongly linked to unconventional electron pairing mechanisms, which significantly deviate from the Cooper pairing observed in traditional Bardeen-Cooper-Schrieffer superconductors. Our results advance the knowledge of Cooper-pair density modulations and the intricate interplay with other symmetry-breaking states in strongly correlated superconductors.

cond-mat.supr-con↗

arXiv2024

Instruction-Driven Fusion of Infrared-Visible Images: Tailoring for Diverse Downstream Tasks

The primary value of infrared and visible image fusion technology lies in applying the fusion results to downstream tasks. However, existing methods face challenges such as increased training complexity and significantly compromised performance of individual tasks when addressing multiple downstream tasks simultaneously. To tackle this, we propose Task-Oriented Adaptive Regulation (T-OAR), an adaptive mechanism specifically designed for multi-task environments. Additionally, we introduce the Task-related Dynamic Prompt Injection (T-DPI) module, which generates task-specific dynamic prompts from user-input text instructions and integrates them into target representations. This guides the feature extraction module to produce representations that are more closely aligned with the specific requirements of downstream tasks. By incorporating the T-DPI module into the T-OAR framework, our approach generates fusion images tailored to task-specific requirements without the need for separate training or task-specific weights. This not only reduces computational costs but also enhances adaptability and performance across multiple tasks. Experimental results show that our method excels in object detection, semantic segmentation, and salient object detection, demonstrating its strong adaptability, flexibility, and task specificity. This provides an efficient solution for image fusion in multi-task environments, highlighting the technology's potential across diverse applications.

cs.CV↗

arXiv2024

Mechanism of magnetic phase transition in correlated magnetic metal: insight into itinerant ferromagnet Fe$_{3-δ}$GeTe$_2$

Developing a comprehensive magnetic theory for correlated itinerant magnets poses challenges due to the difficulty in reconciling both local moments and itinerant electrons. In this work, we investigate the microscopic process of magnetic phase transition in ferromagnetic metal Fe$_{3-δ}$GeTe$_2$. We find that Hund's coupling is crucial for establishing ferromagnetic order. During the ferromagnetic transition, we observe the formation of quasiparticle flat bands and an opposing tendency in spectral weight transfer, primarily between the lower and upper Hubbard bands, across the two spin channels. Moreover, our results indicate that one of the inequivalent Fe sites exhibits Mott physics, while the other Fe site exhibits Hund's physics, attributable to their distinct atomic environments. We suggest that ferromagnetic order reduces spin fluctuations and makes flat bands near the Fermi level more distinct. The hybridization between the distinctly flat bands and other itinerant bands offers a possible way to form heavy fermion behavior in ferromagnets. The complex interactions of competing orders drive correlated magnetic metals to a new frontier for discovering outstanding quantum states.

cond-mat.str-el↗

arXiv2024

"I Came Across a Junk": Understanding Design Flaws of Data Visualization from the Public's Perspective

The visualization community has a rich history of reflecting upon flaws of visualization design, and research in this direction has remained lively until now. However, three main gaps still exist. First, most existing work characterizes design flaws from the perspective of researchers rather than the perspective of general users. Second, little work has been done to infer why these design flaws occur. Third, due to problems such as unclear terminology and ambiguous research scope, a better framework that systematically outlines various design flaws and helps distinguish different types of flaws is desired. To address the above gaps, this work investigated visualization design flaws through the lens of the public, constructed a framework to summarize and categorize the identified flaws, and explored why these flaws occur.

cs.HC↗

arXiv2024

Improved Video VAE for Latent Video Diffusion Model

Variational Autoencoder (VAE) aims to compress pixel data into low-dimensional latent space, playing an important role in OpenAI's Sora and other latent video diffusion generation models. While most of existing video VAEs inflate a pretrained image VAE into the 3D causal structure for temporal-spatial compression, this paper presents two astonishing findings: (1) The initialization from a well-trained image VAE with the same latent dimensions suppresses the improvement of subsequent temporal compression capabilities. (2) The adoption of causal reasoning leads to unequal information interactions and unbalanced performance between frames. To alleviate these problems, we propose a keyframe-based temporal compression (KTC) architecture and a group causal convolution (GCConv) module to further improve video VAE (IV-VAE). Specifically, the KTC architecture divides the latent space into two branches, in which one half completely inherits the compression prior of keyframes from a lower-dimension image VAE while the other half involves temporal compression through 3D group causal convolution, reducing temporal-spatial conflicts and accelerating the convergence speed of video VAE. The GCConv in above 3D half uses standard convolution within each frame group to ensure inter-frame equivalence, and employs causal logical padding between groups to maintain flexibility in processing variable frame video. Extensive experiments on five benchmarks demonstrate the SOTA video reconstruction and generation capabilities of the proposed IV-VAE (https://wpy1999.github.io/IV-VAE/).

cs.CV↗

arXiv2024

Generalizing Hyperedge Expansion for Hyper-relational Knowledge Graph Modeling

By representing knowledge in a primary triple associated with additional attribute-value qualifiers, hyper-relational knowledge graph (HKG) that generalizes triple-based knowledge graph (KG) has been attracting research attention recently. Compared with KG, HKG is enriched with the semantic qualifiers as well as the hyper-relational graph structure. However, to model HKG, existing studies mainly focus on either semantic information or structural information therein, which however fail to capture both simultaneously. To tackle this issue, in this paper, we generalize the hyperedge expansion in hypergraph learning and propose an equivalent transformation for HKG modeling, referred to as TransEQ. Specifically, the equivalent transformation transforms a HKG to a KG, which considers both semantic and structural characteristics. Then an encoder-decoder framework is developed to bridge the modeling research between KG and HKG. In the encoder part, KG-based graph neural networks are leveraged for structural modeling; while in the decoder part, various HKG-based scoring functions are exploited for semantic modeling. Especially, we design the sharing embedding mechanism in the encoder-decoder framework with semantic relatedness captured. We further theoretically prove that TransEQ preserves complete information in the equivalent transformation, and also achieves full expressivity. Finally, extensive experiments on three benchmarks demonstrate the superior performance of TransEQ in terms of both effectiveness and efficiency. On the largest benchmark WikiPeople, TransEQ significantly improves the state-of-the-art models by 15\% on MRR.

cs.AI↗