arXiv subjects

Yulin Pan

Publications and source records attributed to Yulin Pan.

At least 37 records · Page 2Linked to original sources

arXiv2025

Locate, Assign, Refine: Taming Customized Promptable Image Inpainting

Prior studies have made significant progress in image inpainting guided by either text description or subject image. However, the research on inpainting with flexible guidance or control, i.e., text-only, image-only, and their combination, is still in the early stage. Therefore, in this paper, we introduce the multimodal promptable image inpainting project: a new task model, and data for taming customized image inpainting. We propose LAR-Gen, a novel approach for image inpainting that enables seamless inpainting of specific region in images corresponding to the mask prompt, incorporating both the text prompt and image prompt. Our LAR-Gen adopts a coarse-to-fine manner to ensure the context consistency of source image, subject identity consistency, local semantic consistency to the text description, and smoothness consistency. It consists of three mechanisms: (i) Locate mechanism: concatenating the noise with masked scene image to achieve precise regional editing, (ii) Assign mechanism: employing decoupled cross-attention mechanism to accommodate multi-modal guidance, and (iii) Refine mechanism: using a novel RefineNet to supplement subject details. Additionally, to address the issue of scarce training data, we introduce a novel data engine to automatically extract substantial pairs of data consisting of local text prompts and corresponding visual instances from a vast image data, leveraging publicly available pre-trained large models. Extensive experiments and various application scenarios demonstrate the superiority of LAR-Gen in terms of both identity preservation and text semantic consistency.

cs.CV↗

arXiv2025

ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling

We report ACE++, an instruction-based diffusion framework that tackles various image generation and editing tasks. Inspired by the input format for the inpainting task proposed by FLUX.1-Fill-dev, we improve the Long-context Condition Unit (LCU) introduced in ACE and extend this input paradigm to any editing and generation tasks. To take full advantage of image generative priors, we develop a two-stage training scheme to minimize the efforts of finetuning powerful text-to-image diffusion models like FLUX.1-dev. In the first stage, we pre-train the model using task data with the 0-ref tasks from the text-to-image model. There are many models in the community based on the post-training of text-to-image foundational models that meet this training paradigm of the first stage. For example, FLUX.1-Fill-dev deals primarily with painting tasks and can be used as an initialization to accelerate the training process. In the second stage, we finetune the above model to support the general instructions using all tasks defined in ACE. To promote the widespread application of ACE++ in different scenarios, we provide a comprehensive set of models that cover both full finetuning and lightweight finetuning, while considering general applicability and applicability in vertical scenarios. The qualitative analysis showcases the superiority of ACE++ in terms of generating image quality and prompt following ability. Code and models will be available on the project page: https://ali-vilab. github.io/ACE_plus_page/.

cs.CV↗

arXiv2024

StyleBooth: Image Style Editing with Multimodal Instruction

Given an original image, image editing aims to generate an image that align with the provided instruction. The challenges are to accept multimodal inputs as instructions and a scarcity of high-quality training data, including crucial triplets of source/target image pairs and multimodal (text and image) instructions. In this paper, we focus on image style editing and present StyleBooth, a method that proposes a comprehensive framework for image editing and a feasible strategy for building a high-quality style editing dataset. We integrate encoded textual instruction and image exemplar as a unified condition for diffusion model, enabling the editing of original image following multimodal instructions. Furthermore, by iterative style-destyle tuning and editing and usability filtering, the StyleBooth dataset provides content-consistent stylized/plain image pairs in various categories of styles. To show the flexibility of StyleBooth, we conduct experiments on diverse tasks, such as text-based style editing, exemplar-based style editing and compositional style editing. The results demonstrate that the quality and variety of training data significantly enhance the ability to preserve content and improve the overall quality of generated images in editing tasks. Project page can be found at https://ali-vilab.github.io/stylebooth-page/.

cs.CV↗

arXiv2024

Distribution of plastics of various sizes and densities in the global ocean from a 3D Eulerian model

We develop a 3D Eulerian model to study the transport and distribution of microplastics in the global ocean. Among other benefits that will be discussed in the paper, one unique feature of our model is that it takes into consideration the effect of properties of particles (size and density, the former for the first time) to their vertical terminal velocity. With ocean current velocity taken from ECCOv4r4, a dataset generated from a data-assimilated MITgcm reanalysis, our model is integrated for 26 years for particles of different properties with their stationary patterns studied. We find that only low-density particles with sufficient size (e.g. density $900kg/m^3$ with size $\gtrsim 10 μm$) aggregate in the five subtropical gyres observed in previous studies. In contrast, particles of smaller size ($\sim 1 μm$), irrespective of their density, behave like neutrally buoyant particles with a weaker pattern on the surface and a deeper penetration into depth (up to about 1km deep). In addition, we observe seasonal variations of floating particle concentration on the ocean surface, which reasonably agree with the satellite observation by Cyclone Global Navigation Satellite System (CYGNSS) in terms of the phase of the variation. We find that the seasonal variation of the surface particle concentration correlates well with the variation of the mixing layer (ML) depth globally, due to an almost uniform vertical distribution of particles in the ML with total amount of particles conserved.

physics.ao-ph↗

arXiv2024

CFD-based design optimization of a 5 kW ducted hydrokinetic turbine with practical constraints

Ducted hydrokinetic turbines enhance energy-harvesting efficiency by better conditioning the flow to the blades, which may yield higher power output than conventional freestream turbines for the same reference area. In this work, we present a ducted hydrokinetic turbine design obtained by simultaneously optimizing the duct, blade, and hub geometries. Our optimization framework combines a CFD solver, an adjoint solver, and a gradient-based optimizer to efficiently explore a large design space, together with a feature-based parameterization method to handle the complex geometry. Practical geometrical constraints ensure the manufacturability of the duct in terms of a minimum thickness and the housing of a 5 kW generator within the hub. The optimization converges to a short, thin duct with a rounded leading edge and an elongated hub protruding the duct inlet. The optimized ducted turbine achieves up to 50% efficiency when evaluated by RANS/URANS solvers despite a bulky hub, outperforming the 45% efficiency of the freestream Bahaj turbine featuring the same hub. This work showcases the effectiveness of CFD-based optimization in advancing ducted turbine designs and demonstrates the hydrodynamic benefits of a ducted configuration, paving the way for future research and real-world applications.

physics.flu-dyn↗

arXiv2024

ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer

Diffusion models have emerged as a powerful generative technology and have been found to be applicable in various scenarios. Most existing foundational diffusion models are primarily designed for text-guided visual generation and do not support multi-modal conditions, which are essential for many visual editing tasks. This limitation prevents these foundational diffusion models from serving as a unified model in the field of visual generation, like GPT-4 in the natural language processing field. In this work, we propose ACE, an All-round Creator and Editor, which achieves comparable performance compared to those expert models in a wide range of visual generation tasks. To achieve this goal, we first introduce a unified condition format termed Long-context Condition Unit (LCU), and propose a novel Transformer-based diffusion model that uses LCU as input, aiming for joint training across various generation and editing tasks. Furthermore, we propose an efficient data collection approach to address the issue of the absence of available training data. It involves acquiring pairwise images with synthesis-based or clustering-based pipelines and supplying these pairs with accurate textual instructions by leveraging a fine-tuned multi-modal large language model. To comprehensively evaluate the performance of our model, we establish a benchmark of manually annotated pairs data across a variety of visual generation tasks. The extensive experimental results demonstrate the superiority of our model in visual generation fields. Thanks to the all-in-one capabilities of our model, we can easily build a multi-modal chat system that responds to any interactive request for image creation using a single model to serve as the backend, avoiding the cumbersome pipeline typically employed in visual agents. Code and models will be available on the project page: https://ali-vilab.github.io/ace-page/.

cs.CV↗

arXiv2024

Role of triad interactions in spectral evolution of surface gravity waves in deep water

It is generally accepted that the evolution of deep-water surface gravity wave spectrum is governed by quartet resonant and quasi-resonant interactions. However, it has also been reported in both experimental and computational studies that non-resonant triad interactions can play a role, e.g., generation of bound waves. In this study, we investigate the effects of triad and quartet interactions on the spectral evolution, by numerically tracking the contributions from quadratic and cubic terms in the dynamical equation. In a finite time interval, we find that the contribution from triad interactions follows the trend of that from quartet resonances (with comparable magnitude) for most wavenumbers, except that it peaks at low wavenumbers with very low initial energy. This result reveals two effects of triad interactions: (1) the non-resonant triad interactions can be connected to form quartet resonant interactions (hence exhibiting the comparable trend), which is a reflection of the normal form transformation applied in wave turbulence theory of surface gravity waves. (2) the triad interactions can fill energy into the low energy portion of the spectrum (low wavenumber part in this case) on a very fast time scale, with energy distributed in both bound and free modes at the same wavenumber. We further analyze the latter mechanism using a simple model with two initially active modes in the wavenumber domain. Analytical formulae are provided to describe the distribution of energy in free and bound modes with numerical validations.

physics.flu-dyn↗

arXiv2024

Transition from weak turbulence to collapse turbulence regimes in MMT model

It is well known that wave collapses can emerge from the focusing one-dimensional (1-D) Majda-McLaughlin-Tabak (MMT) model as a result of modulational instability. However, how these wave collapses affect the spectral properties and statistics of the wave field has not been adequately studied. We undertake this task by simulating the forced-dissipated 1-D MMT model over a range of forcing amplitudes. Our results show that when the forcing is weak, the spectrum agrees well with the prediction by wave turbulence theory with few collapses in the field. As the forcing strength increases, we see an increase in the occurrence of collapses, together with a transition from a power-law spectrum to an exponentially decaying spectrum. Through a spectral decomposition, we find that the exponential spectrum is dominated by the wave collapse component in the non-integrable MMT model, which is in analogy to a soliton gas in integrable turbulence.

nlin.PS↗

arXiv2024

Energy transfer for solutions to the nonlinear Schrödinger equation on irrational tori

We analyze the energy transfer for solutions to the defocusing cubic nonlinear Schrödinger (NLS) initial value problem on 2D irrational tori. Moreover we complement the analytic study with numerical experimentation. As a biproduct of our investigation we also prove that the quasi-resonant part of the NLS initial value problem we consider, in both the focusing and defocusing case, is globally well-posed for initial data of finite mass.

math.AP↗

arXiv2024

Verification of Wave Turbulence Theory in the Kinetic Limit

Using the 1D Majda-McLaughlin-Tabak model as an example, we develop numerical experiments to study the validity of the Wave Kinetic Equation (WKE) at the kinetic limit (i.e., small nonlinearity and large domain). We show that the dynamics converges to the WKE prediction, in terms of the closure model and energy flux, when the kinetic limit is approached. When the kinetic limit is combined with a process of widening the inertial range, the theoretical Kolmogorov constant can be recovered numerically to a very high precision.

physics.flu-dyn↗

arXiv2023

SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing

Image diffusion models have been utilized in various tasks, such as text-to-image generation and controllable image synthesis. Recent research has introduced tuning methods that make subtle adjustments to the original models, yielding promising results in specific adaptations of foundational generative diffusion models. Rather than modifying the main backbone of the diffusion model, we delve into the role of skip connection in U-Net and reveal that hierarchical features aggregating long-distance information across encoder and decoder make a significant impact on the content and quality of image generation. Based on the observation, we propose an efficient generative tuning framework, dubbed SCEdit, which integrates and edits Skip Connection using a lightweight tuning module named SC-Tuner. Furthermore, the proposed framework allows for straightforward extension to controllable image synthesis by injecting different conditions with Controllable SC-Tuner, simplifying and unifying the network design for multi-condition inputs. Our SCEdit substantially reduces training parameters, memory usage, and computational expense due to its lightweight tuners, with backward propagation only passing to the decoder blocks. Extensive experiments conducted on text-to-image generation and controllable image synthesis tasks demonstrate the superiority of our method in terms of efficiency and performance. Project page: \url{https://scedit.github.io/}

cs.CV↗

arXiv2023

Energy cascade in the Garrett-Munk spectrum of internal gravity waves

We study the spectral energy transfer due to wave-triad interactions in the Garrett-Munk spectrum of internal gravity waves (IGWs) based on a numerical evaluation of the collision integral in the wave kinetic equation. Our numerical evaluation builds on the reduction of the collision integral on the resonant manifold for a horizontally isotropic spectrum. We directly evaluate the downscale energy flux available for ocean mixing, whose value is in close agreement with the empirical finescale parameterization. We further decompose the energy transfer into contributions from different mechanisms, including local interactions and three types of nonlocal interactions, namely parametric subharmonic instability (PSI), elastic scattering (ES) and induced diffusion (ID). Through analysis on the role of each type of interaction, we resolve two long-standing paradoxes regarding the mechanism for forward cascade in frequency and zero ID flux for GM76 spectrum. In addition, our analysis estimates the contribution of each mechanism to the energy transfer in each spectral direction, and reveals new understanding of the importance of local interactions and ES in the energy transfer.

physics.flu-dyn↗

arXiv2023

A generalized likelihood-weighted optimal sampling algorithm for rare-event probability quantification

In this work, we introduce a new acquisition function for sequential sampling to efficiently quantify rare-event statistics of an input-to-response (ItR) system with given input probability and expensive function evaluations. Our acquisition is a generalization of the likelihood-weighted (LW) acquisition that was initially designed for the same purpose and then extended to many other applications. The improvement in our acquisition comes from the generalized form with two additional parameters, by varying which one can target and address two weaknesses of the original LW acquisition: (1) that the input space associated with rare-event responses is not sufficiently stressed in sampling; (2) that the surrogate model (generated from samples) may have significant deviation from the true ItR function, especially for cases with complex ItR function and limited number of samples. In addition, we develop a critical procedure in Monte-Carlo discrete optimization of the acquisition function, which achieves orders of magnitude acceleration compared to existing approaches for such type of problems. The superior performance of our new acquisition to the original LW acquisition is demonstrated in a number of test cases, including some cases that were designed to show the effectiveness of the original LW acquisition. We finally apply our method to an engineering example to quantify the rare-event roll-motion statistics of a ship in a random sea.

cs.RO↗

arXiv2023

CFD-based Design Optimization of Ducted Hydrokinetic Turbines

Hydrokinetic turbines extract kinetic energy from moving water to generate renewable electricity, thus contributing to sustainable energy production and reducing reliance on fossil fuels. It has been hypothesized that a duct can accelerate and condition the fluid flow passing the turbine blades, improving the overall energy extraction efficiency. However, no substantial evidence has been provided so far for hydrokinetic turbines. To investigate this problem, we perform a CFD-based optimization study with a blade-resolved Reynolds-averaged Navier--Stokes (RANS) solver to explore the design of a ducted hydrokinetic turbine that maximizes the efficiency of energy extraction. To handle the high-dimensional design space of the blade and duct geometry, we use a gradient-based optimization approach where the gradients are computed using the adjoint method. The final design is re-evaluated through higher-fidelity unsteady RANS (URANS) simulations. Our optimized ducted turbine achieves an efficiency of about 54% over a range of operating conditions, higher than the typical 46% efficiency of unducted turbines such as the well-known Bahaj model.

physics.flu-dyn↗

arXiv2023

An adaptive multi-fidelity sampling framework for safety analysis of connected and automated vehicles

Testing and evaluation are expensive but critical steps in the development of connected and automated vehicles (CAVs). In this paper, we develop an adaptive sampling framework to efficiently evaluate the accident rate of CAVs, particularly for scenario-based tests where the probability distribution of input parameters is known from the Naturalistic Driving Data. Our framework relies on a surrogate model to approximate the CAV performance and a novel acquisition function to maximize the benefit (information to accident rate) of the next sample formulated through an information-theoretic consideration. In addition to the standard application with only a single high-fidelity model of CAV performance, we also extend our approach to the bi-fidelity context where an additional low-fidelity model can be used at a lower computational cost to approximate the CAV performance. Accordingly, for the second case, our approach is formulated such that it allows the choice of the next sample in terms of both fidelity level (i.e., which model to use) and sampling location to maximize the benefit per cost. Our framework is tested in a widely-considered two-dimensional cut-in problem for CAVs, where Intelligent Driving Model (IDM) with different time resolutions are used to construct the high and low-fidelity models. We show that our single-fidelity method outperforms the existing approach for the same problem, and the bi-fidelity method can further save half of the computational cost to reach a similar accuracy in estimating the accident rate.

cs.RO↗

arXiv2023

On the time scales of spectral evolution of nonlinear waves

As presented in Annenkov & Shrira (2009), when a surface gravity wave field is subjected to an abrupt perturbation of external forcing, its spectrum evolves on a ``fast'' dynamic time scale of $O(\varepsilon^{-2})$, with $\varepsilon$ a measure of wave steepness. This observation poses a challenge to wave turbulence theory that predicts an evolution with a kinetic time scale of $O(\varepsilon^{-4})$. We revisit this unresolved problem by studying the same situation in the context of a one-dimensional Majda-McLaughlin-Tabak (MMT) equation with gravity wave dispersion relation. Our results show that the kinetic and dynamic time scales can both be realised, with the former and latter occurring for weaker and stronger forcing perturbations, respectively. The transition between the two regimes corresponds to a critical forcing perturbation, with which the spectral evolution time scale drops to the same order as the linear wave period (of some representative mode). Such fast spectral evolution is mainly induced by a far-from-stationary state after a sufficiently strong forcing perturbation is applied. We further develop a set-based interaction analysis to show that the inertial-range modal evolution in the studied cases is dominated by their (mostly non-local) interactions with the low-wavenumber ``condensate'' induced by the forcing perturbation. The results obtained in this work should be considered to provide significant insight into the original gravity wave problem.

physics.flu-dyn↗

arXiv2023

VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrieval

Many recent studies leverage the pre-trained CLIP for text-video cross-modal retrieval by tuning the backbone with additional heavy modules, which not only brings huge computational burdens with much more parameters, but also leads to the knowledge forgetting from upstream models. In this work, we propose the VoP: Text-Video Co-operative Prompt Tuning for efficient tuning on the text-video retrieval task. The proposed VoP is an end-to-end framework with both video & text prompts introducing, which can be regarded as a powerful baseline with only 0.1% trainable parameters. Further, based on the spatio-temporal characteristics of videos, we develop three novel video prompt mechanisms to improve the performance with different scales of trainable parameters. The basic idea of the VoP enhancement is to model the frame position, frame context, and layer function with specific trainable prompts, respectively. Extensive experiments show that compared to full fine-tuning, the enhanced VoP achieves a 1.4% average R@1 gain across five text-video retrieval benchmarks with 6x less parameter overhead. The code will be available at https://github.com/bighuang624/VoP.

cs.CV↗

arXiv2023

Scanning Only Once: An End-to-end Framework for Fast Temporal Grounding in Long Videos

Video temporal grounding aims to pinpoint a video segment that matches the query description. Despite the recent advance in short-form videos (\textit{e.g.}, in minutes), temporal grounding in long videos (\textit{e.g.}, in hours) is still at its early stage. To address this challenge, a common practice is to employ a sliding window, yet can be inefficient and inflexible due to the limited number of frames within the window. In this work, we propose an end-to-end framework for fast temporal grounding, which is able to model an hours-long video with \textbf{one-time} network execution. Our pipeline is formulated in a coarse-to-fine manner, where we first extract context knowledge from non-overlapped video clips (\textit{i.e.}, anchors), and then supplement the anchors that highly response to the query with detailed content knowledge. Besides the remarkably high pipeline efficiency, another advantage of our approach is the capability of capturing long-range temporal correlation, thanks to modeling the entire video as a whole, and hence facilitates more accurate grounding. Experimental results suggest that, on the long-form video datasets MAD and Ego4d, our method significantly outperforms state-of-the-arts, and achieves \textbf{14.6$\times$} / \textbf{102.8$\times$} higher efficiency respectively. Project can be found at \url{https://github.com/afcedf/SOONet.git}.

cs.CV↗