arXiv subjects

Yifan Du

Publications and source records attributed to Yifan Du.

33 records · Page 2Linked to original sources

arXiv2024

Exploring the Design Space of Visual Context Representation in Video MLLMs

Video Multimodal Large Language Models (MLLMs) have shown remarkable capability of understanding the video semantics on various downstream tasks. Despite the advancements, there is still a lack of systematic research on visual context representation, which refers to the scheme to select frames from a video and further select the tokens from a frame. In this paper, we explore the design space for visual context representation, and aim to improve the performance of video MLLMs by finding more effective representation schemes. Firstly, we formulate the task of visual context representation as a constrained optimization problem, and model the language modeling loss as a function of the number of frames and the number of embeddings (or tokens) per frame, given the maximum visual context window size. Then, we explore the scaling effects in frame selection and token selection respectively, and fit the corresponding function curve by conducting extensive empirical experiments. We examine the effectiveness of typical selection strategies and present empirical findings to determine the two factors. Furthermore, we study the joint effect of frame selection and token selection, and derive the optimal formula for determining the two factors. We demonstrate that the derived optimal settings show alignment with the best-performed results of empirical experiments. Our code and model are available at: https://github.com/RUCAIBox/Opt-Visor.

cs.CV↗

arXiv2024

Towards Event-oriented Long Video Understanding

With the rapid development of video Multimodal Large Language Models (MLLMs), numerous benchmarks have been proposed to assess their video understanding capability. However, due to the lack of rich events in the videos, these datasets may suffer from the short-cut bias that the answers can be deduced from a few frames, without the need to watch the entire video. To address this issue, we introduce Event-Bench, an event-oriented long video understanding benchmark built on existing datasets and human annotations. Event-Bench includes six event-related tasks and 2,190 test instances to comprehensively evaluate video event understanding ability. Additionally, we propose Video Instruction Merging~(VIM), a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions, addressing the scarcity of human-annotated, event-intensive data. Extensive experiments show that the best-performing model, GPT-4o, achieves an overall accuracy of 53.33, significantly outperforming the best open-source model by 41.42%. Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench. All code, data, and models are publicly available at https://github.com/RUCAIBox/Event-Bench.

cs.CV↗

arXiv2024

Bayesian data-driven discovery of partial differential equations with variable coefficients

The discovery of Partial Differential Equations (PDEs) is an essential task for applied science and engineering. However, data-driven discovery of PDEs is generally challenging, primarily stemming from the sensitivity of the discovered equation to noise and the complexities of model selection. In this work, we propose an advanced Bayesian sparse learning algorithm for PDE discovery with variable coefficients, predominantly when the coefficients are spatially or temporally dependent. Specifically, we apply threshold Bayesian group Lasso regression with a spike-and-slab prior (tBGL-SS) and leverage a Gibbs sampler for Bayesian posterior estimation of PDE coefficients. This approach not only enhances the robustness of point estimation with valid uncertainty quantification but also relaxes the computational burden from Bayesian inference through the integration of coefficient thresholds as an approximate MCMC method. Moreover, from the quantified uncertainties, we propose a Bayesian total error bar criteria for model selection, which outperforms classic metrics including the root mean square and the Akaike information criterion. The capability of this method is illustrated by the discovery of several classical benchmark PDEs with spatially or temporally varying coefficients from solution data obtained from the reference simulations. In the experiments, we show that the tBGL-SS method is more robust than the baseline methods under noisy environments and provides better model selection criteria along the regularization path.

stat.ML↗

arXiv2023

Evaluating Object Hallucination in Large Vision-Language Models

Inspired by the superior language abilities of large language models (LLM), large vision-language models (LVLM) have been recently explored by integrating powerful LLMs for improving the performance on complex multimodal tasks. Despite the promising progress on LVLMs, we find that LVLMs suffer from the hallucination problem, i.e. they tend to generate objects that are inconsistent with the target images in the descriptions. To investigate it, this work presents the first systematic study on object hallucination of LVLMs. We conduct the evaluation experiments on several representative LVLMs, and show that they mostly suffer from severe object hallucination issue. We further discuss that the visual instructions may influence the hallucination, and find that: objects that frequently occur in the visual instructions or co-occur with the image objects, are obviously prone to be hallucinated by LVLMs. Besides, we find that existing evaluation methods might be affected by the input instructions and generation styles of LVLMs. Thus, we further design an improved evaluation method for object hallucination by proposing a polling-based query method called POPE. Experiment results demonstrate that our POPE can evaluate the object hallucination in a more stable and flexible way. Our codes and data are publicly available at https://github.com/RUCAIBox/POPE.

cs.CV↗

arXiv2023

Learning to Imagine: Visually-Augmented Natural Language Generation

People often imagine relevant scenes to aid in the writing process. In this work, we aim to utilize visual information for composition in the same manner as humans. We propose a method, LIVE, that makes pre-trained language models (PLMs) Learn to Imagine for Visuallyaugmented natural language gEneration. First, we imagine the scene based on the text: we use a diffusion model to synthesize high-quality images conditioned on the input texts. Second, we use CLIP to determine whether the text can evoke the imagination in a posterior way. Finally, our imagination is dynamic, and we conduct synthesis for each sentence rather than generate only one image for an entire paragraph. Technically, we propose a novel plug-and-play fusion layer to obtain visually-augmented representations for each text. Our vision-text fusion layer is compatible with Transformerbased architecture. We have conducted extensive experiments on four generation tasks using BART and T5, and the automatic results and human evaluation demonstrate the effectiveness of our proposed method. We will release the code, model, and data at the link: https://github.com/RUCAIBox/LIVE.

cs.CL↗

arXiv2023

Zero-shot Visual Question Answering with Language Model Feedback

In this paper, we propose a novel language model guided captioning approach, LAMOC, for knowledge-based visual question answering (VQA). Our approach employs the generated captions by a captioning model as the context of an answer prediction model, which is a Pre-trained Language model (PLM). As the major contribution, we leverage the guidance and feedback of the prediction model to improve the capability of the captioning model. In this way, the captioning model can become aware of the task goal and information need from the PLM. To develop our approach, we design two specific training stages, where the first stage adapts the captioning model to the prediction model (selecting more suitable caption propositions for training) and the second stage tunes the captioning model according to the task goal (learning from feedback of the PLM). Extensive experiments demonstrate the effectiveness of the proposed approach on the knowledge-based VQA task. Specifically, on the challenging A-OKVQA dataset, LAMOC outperforms several competitive zero-shot methods and even achieves comparable results to a fine-tuned VLP model. Our code is publicly available at https://github.com/RUCAIBox/LAMOC.

cs.CV↗

arXiv2022

State estimation in minimal turbulent channel flow: A comparative study of 4DVar and PINN

The state of turbulent, minimal-channel flow is estimated from spatio-temporal sparse observations of the velocity, using both a physics-informed neural network (PINN) and adjoint-variational data assimilation (4DVar). The performance of PINN is assessed against the benchmark results from 4DVar. The PINN is efficient to implement, takes advantage of automatic differentiation to evaluate the governing equations, and does not require the development of an adjoint model. In addition, the flow evolution is expressed in terms of the network parameters which have a far smaller dimension than the predicted trajectory in state space or even just the initial condition of the flow. Provided adequate observations, network architecture and training, the PINN can yield satisfactory estimates of the the flow field, both for the missing velocity data and the entirely unobserved pressure field. However, accuracy depends on the network architecture, and the dependence is not known a priori. In comparison to 4DVar estimation which becomes progressively more accurate over the observation horizon, the PINN predictions are generally less accurate and maintain the same level of errors throughout the assimilation time window. Another notable distinction is the capacity to accurately forecast the flow evolution: while the 4DVar prediction depart from the true flow state gradually and according to the Lyapunov exponent, the PINN is entirely inaccurate immediately beyond the training time horizon unless re-trained. Most importantly, while 4DVar satisfies the discrete form of the governing equations point-wise to machine precision, in PINN the equations are only satisfied in an $L^2$ sense.

physics.flu-dyn↗

arXiv2022

A Survey of Vision-Language Pre-Trained Models

As transformer evolves, pre-trained models have advanced at a breakneck pace in recent years. They have dominated the mainstream techniques in natural language processing (NLP) and computer vision (CV). How to adapt pre-training to the field of Vision-and-Language (V-L) learning and improve downstream task performance becomes a focus of multimodal learning. In this paper, we review the recent progress in Vision-Language Pre-Trained Models (VL-PTMs). As the core content, we first briefly introduce several ways to encode raw images and texts to single-modal embeddings before pre-training. Then, we dive into the mainstream architectures of VL-PTMs in modeling the interaction between text and image representations. We further present widely-used pre-training tasks, and then we introduce some common downstream tasks. We finally conclude this paper and present some promising research directions. Our survey aims to provide researchers with synthesis and pointer to related research.

cs.CV↗

arXiv2021

Ensemble-variational assimilation of statistical data in large eddy simulation

A non-intrusive data assimilation methodology is developed to improve the statistical predictions of large-eddy simulations (LES). The ensemble-variational (EnVar) approach aims to minimize a cost function that is defined as the discrepancy between LES predictions and reference statistics from experiments or, in the present demonstration, independent direct numerical simulations (DNS). This methodology is applied to adjust the Smagorinsky subgrid model and obtain data assimilated LES (DA-LES) which accurately estimate the statistics of turbulent channel flow. To separately control the mean and fluctuations of the modeled subgrid tensor, and ultimately the first- and second-order flow statistics, two types of model corrections are considered. The first one optimizes the wall-normal profile of the Smagorinsky coefficient, while the second one introduces an adjustable steady forcing in the momentum equations to independently act on the mean flow. Using these two elements, the data assimilation procedure can satisfactorily modify the subgrid model and accurately recover reference flow statistics. The retrieved subgrid model significantly outperforms more elaborate baseline models such as dynamic and mixed models, in a posteriori testing. The robustness of the present data assimilation methodology is assessed by changing the Reynolds number and considering grid resolutions that are away from usual recommendations. Taking advantage of the stochastic formulation of EnVar, the developed framework also provides the uncertainty of the retrieved model.

physics.flu-dyn↗

arXiv2021

Evolutional Deep Neural Network

The notion of an Evolutional Deep Neural Network (EDNN) is introduced for the solution of partial differential equations (PDE). The parameters of the network are trained to represent the initial state of the system only, and are subsequently updated dynamically, without any further training, to provide an accurate prediction of the evolution of the PDE system. In this framework, the network parameters are treated as functions with respect to the appropriate coordinate and are numerically updated using the governing equations. By marching the neural network weights in the parameter space, EDNN can predict state-space trajectories that are indefinitely long, which is difficult for other neural network approaches. Boundary conditions of the PDEs are treated as hard constraints, are embedded into the neural network, and are therefore exactly satisfied throughout the entire solution trajectory. Several applications including the heat equation, the advection equation, the Burgers equation, the Kuramoto Sivashinsky equation and the Navier-Stokes equations are solved to demonstrate the versatility and accuracy of EDNN. The application of EDNN to the incompressible Navier-Stokes equation embeds the divergence-free constraint into the network design so that the projection of the momentum equation to solenoidal space is implicitly achieved. The numerical results verify the accuracy of EDNN solutions relative to analytical and benchmark numerical solutions, both for the transient dynamics and statistics of the system.

physics.comp-ph↗

arXiv2020

RotEqNet: Rotation-Equivariant Network for Fluid Systems with Symmetric High-Order Tensors

In the recent application of scientific modeling, machine learning models are largely applied to facilitate computational simulations of fluid systems. Rotation symmetry is a general property for most symmetric fluid systems. However, in general, current machine learning methods have no theoretical way to guarantee rotational symmetry. By observing an important property of contraction and rotation operation on high-order symmetric tensors, we prove that the rotation operation is preserved via tensor contraction. Based on this theoretical justification, in this paper, we introduce Rotation-Equivariant Network (RotEqNet) to guarantee the property of rotation-equivariance for high-order tensors in fluid systems. We implement RotEqNet and evaluate our claims through four case studies on various fluid systems. The property of error reduction and rotation-equivariance is verified in these case studies. Results from the comparative study show that our method outperforms conventional methods, which rely on data augmentation.

cs.CE↗

arXiv2018

Optimizing Multi-hop Mechanism for the Long Range Wide Area Network

Long Range (LoRa) is a modem technology for wireless communication in the Internet of Things (IoT), which trades off low data-rate for low power consumption. Long Range Wide Area Network (LoRaWAN) has an open specification that determines the Medium Access Control (MAC) protocol for LoRa networks. This paper contributes a simulation module in Network Simulator 3 (ns-3) to evaluate the limitation of LoRaWAN in various metrics. The result shows the performance of LoRaWAN in a simulation environment and a large scale scenario, indicating the feature of flexible propagation range on physical layer, and limited channel capacity on the data-link layer. The feasibility of optimized multi-hop in LoRaWAN is analyzed. Based on the simulation result, a multi-hop mechanism driven by spreading factor is proposed, not only to extend the network coverage but to reduce the channel load. The novel approach takes benefit of the dynamic feature of LoRa and estimates optimized forwarding hops with parameter configuration. The result shows that Time on Air can be significantly decreased.

cs.NI↗

arXiv2018

Long Range Wide Area Network: A Simulation Module for ns-3

Long Range (LoRa) is a wireless communication technology for the Internet of Things (IoT), able to provide wide coverage networking to devices with low power consumption and low data-rate. This paper~proposes and describes a simulation module of LoRa wireless networks, based on models of the LoRa physical and data-link layers, in order to enable simulation-based evaluation of LoRa performance in large-scale networks. The physical layer model uses path loss parameters estimated by way of real measurements performed at different scenarios. The data-link layer model relies on an Aloha-like medium access protocol. Both models for the physical and data-link layers are implemented in a ns-3 module. Behavior and performance of this module are studied by way of a set of experiments that show the consistency of ns-3 simulation and measurements on a real LoRa testbed. Finally, an evaluation of the simulation module scope shows that the time complexity of this module is exponential.

cs.NI↗

arXiv2018

Trilateration-based Multi-hop for the Long Range Wide Area Network

Long Range Wide Area Network (LoRaWAN) has an open specification that defines the Medium Access Control (MAC) protocol for LoRa networks, mainly based on pure-Aloha with beacon option. The LoRaWAN has a star of star topology where end-devices are directly linked to the gateway and up-link packets are primary traffic. This paper~contributes an approach of multi-hop in LoRaWAN. The potential benefits of multi-hop in LoRaWAN are investigated. A RSSI-based trilateration algorithm for graph construction is proposed. The shortest path for multi-hop and forwarding mechanism without local beacon is designed and implemented in simulation. This novel approach is tested in ns-3 simulator and different scenarios are run. Results indicate that multi-hop can decrease the energy consumption of LoRaWAN, by reducing Time on Air and TX power.

cs.NI↗

arXiv2017

Turbulence Generation from a stochastic wavelet model

This research presents a new turbulence generation method based on stochastic wavelets and tests its various properties in both homogeneous and inhomogeneous turbulence. Turbulence field can be generated with less basis compared to previous synthetic Fourier methods. Adaptive generation of inhomogeneous turbulence is achieved by scale reduction algorithm and lead to smaller computation cost. The generated turbulence shows good agreement with input data and theoretical results.

math.NA↗