arXiv subjects

Joshua Romero

Publications and source records attributed to Joshua Romero.

8 recordsLinked to original sources

arXiv2026

GPU-Accelerated Simulations of Moving Boundary Problems and Fluid-Structure Interaction at Extreme Scales

Computational fluid dynamics and fluid-structure interaction simulations involving moving and deforming bodies is extremely hard. In this work, we present a graphical processing unit (GPU) optimized implementation of the sharp-interface immersed boundary method. The method allows performing simulation around complex stationary as well as moving bodies on a Cartesian grid. We base our implementation on the ViCar3D framework and make use of OpenACC, CUDA, NCCL and MPI. We test the implementation across grid sizes ranging from O(10million) to O(1billion) points and achieved a 20X speedup compared to existing CPU implementation. We next present our multi-GPU implementation by utilizing CUDA streams and NCCL communicators. This enables us to obtain a >90% strong and weak scaling efficiencies. Next we demonstrate the capability of the developed software to simulate a turbulent fluid flow and coupled fluid-structure interaction in flapping bat wing in flight at Re=5000.

physics.comp-ph↗

arXiv2026

A fast incompressible Navier-Stokes solver for non-uniform grids

We present a scalable incompressible Navier--Stokes solver for three-dimensional Cartesian grids with non-uniform spacing. A direct tensor-product--Thomas method solves the constant-coefficient Poisson and Helmholtz equations arising from pressure projection and implicit diffusion, without approximate factorization. On uniform grids, the method recovers the classical eigenfunction-expansion method evaluated with fast Fourier transforms (FFTs). On stretched grids, diagonal scaling symmetrizes the one-dimensional Laplace operators, and the resulting numerical eigenbasis transforms are evaluated as general matrix--matrix multiplications (GEMMs). FFTs and GEMMs can be selected independently in each diagonalized direction while retaining the pencil decomposition, collective transposes, and tridiagonal machinery of an established FFT-based solver. The elliptic solver is verified to round-off accuracy, and the complete flow solver is validated against benchmark flows. Against geometric multigrid and block cyclic reduction with FFT diagonalization, the present method achieves the lowest time-to-solution among the tested approaches; at fixed grid dimensions, its cost is insensitive to grid stretching. CPU and multi-GPU tests show that GEMM-rich variants attain higher strong-scaling efficiency by better amortizing communication. On a single GPU, the fully GEMM-based variant increases the Poisson cost by $2.8\times$ but the complete Navier--Stokes step cost by only $1.8\times$. Weak scaling exposes the trade-off: dense-transform costs grow with the global transform dimension, whereas non-uniform meshes can reduce the required number of grid points. The resulting open-source solver, \texttt{CaNS-EIGEN}, extends an FFT-based Navier--Stokes solver to large-scale simulations on grids stretched in multiple directions.

physics.comp-ph↗

arXiv2025

A GPU-Accelerated Sharp Interface Immersed Boundary Solver for Large Scale Flow Simulations

Immersed boundary methods (IBMs) facilitate the simulation of flows around stationary, moving, and deforming bodies on Cartesian grids. However, extending these simulations to the large grid sizes required for realistic flow problems remains a significant computational challenge. In this work, we present the implementation and acceleration of ViCar3D, a sharp-interface immersed boundary solver, on graphical processing units (GPUs). We utilize OpenACC, CUDA Fortran and MPI to reprogram \emph{ViCar3D}, a sharp-interface immersed boundary solver, on multi-GPU architectures. Verification and scalability studies are performed for two benchmark cases: two-dimensional flow past a circular cylinder and direct numerical simulation (DNS) of flow past a finite rectangular wing. For the latter, we observe an approximately 20X speedup (node-to-node comparison) relative to the CPU-based implementation. The GPU-accelerated solver is capable of simulating complex 3D flows with up to 200 million mesh points on a single node equipped with four GPUs. Strong and weak scaling tests demonstrate maximum scaling efficiencies of 92\% and 93\%, respectively, on multi-GPU systems. We further test the code to simulate fluid flow past complex-shaped single-body and multi-body cases.

physics.flu-dyn↗

arXiv2021

One-point statistics for turbulent pipe flow up to $Re_τ \approx 6000$

We study turbulent flows in a smooth straight pipe of circular cross--section up to $Re_τ \approx 6000$ using direct--numerical-simulation (DNS) of the Navier--Stokes equations. The DNS results highlight systematic deviations from Prandtl friction law, amounting to about $2\%$, which would extrapolate to about $4\%$ at extreme Reynolds numbers. Data fitting of the DNS friction coefficient yields an estimated von Kármán constant $k \approx 0.387$, which nicely fits the mean velocity profile, and which supports universality of canonical wall-bounded flows. The same constant also applies to the pipe centerline velocity, thus providing support for the claim that the asymptotic state of pipe flow at extreme Reynolds numbers should be plug flow. At the Reynolds numbers under scrutiny, no evidence for saturation of the logarithmic growth of the inner peak of the axial velocity variance is found. Although no outer peak of the velocity variance directly emerges in our DNS, we provide strong evidence that it should appear at $Re_τ \gtrsim 10^4$, as a result of turbulence production exceeding dissipation over a large part of the outer wall layer, thus invalidating the classical equilibrium hypothesis.

physics.flu-dyn↗

arXiv2021

FSEI-GPU: GPU accelerated simulations of the fluid-structure-electrophysiology interaction in the left heart

The reliability of cardiovascular computational models depends on the accurate solution of the hemodynamics, the realistic characterization of the hyperelastic and electric properties of the tissues along with the correct description of their interaction. The resulting fluid-structure-electrophysiology interaction (FSEI) thus requires an immense computational power, usually available in large supercomputing centers, and requires long time to obtain results even if multi-CPU processors are used (MPI acceleration). In recent years, graphics processing units (GPUs) have emerged as a convenient platform for high performance computing, as they allow for considerable reductions of the time-to-solution. This approach is particularly appealing if the tool has to support medical decisions that require solutions within reduced times and possibly obtained by local computational resources. Accordingly, our multi-physics solver has been ported to GPU architectures using CUDA Fortran to tackle fast and accurate hemodynamics simulations of the human heart without resorting to large-scale supercomputers. This work describes the use of CUDA to accelerate the FSEI on heterogeneous clusters, where both the CPUs and GPUs are used in synergistically with minor modifications of the original source code. The resulting GPU accelerated code solves a single heartbeat within a few hours (from three to ten depending on the grid resolution) running on premises computing facility made of few GPU cards, which can be easily installed in a medical laboratory or in a hospital, thus opening towards a systematic computational fluid dynamics (CFD) aided diagnostic.

physics.flu-dyn↗

arXiv2019

Exascale Deep Learning for Scientific Inverse Problems

We introduce novel communication strategies in synchronous distributed Deep Learning consisting of decentralized gradient reduction orchestration and computational graph-aware grouping of gradient tensors. These new techniques produce an optimal overlap between computation and communication and result in near-linear scaling (0.93) of distributed training up to 27,600 NVIDIA V100 GPUs on the Summit Supercomputer. We demonstrate our gradient reduction techniques in the context of training a Fully Convolutional Neural Network to approximate the solution of a longstanding scientific inverse problem in materials imaging. The efficient distributed training on a dataset size of 0.5 PB, produces a model capable of an atomically-accurate reconstruction of materials, and in the process reaching a peak performance of 2.15(4) EFLOPS$_{16}$.

cs.LG↗

arXiv2019

A Performance Study of the 2D Ising Model on GPUs

The simulation of the two-dimensional Ising model is used as a benchmark to show the computational capabilities of Graphic Processing Units (GPUs). The rich programming environment now available on GPUs and flexible hardware capabilities allowed us to quickly experiment with several implementation ideas: a simple stencil-based algorithm, recasting the stencil operations into matrix multiplies to take advantage of Tensor Cores available on NVIDIA GPUs, and a highly optimized multi-spin coding approach. Using the managed memory API available in CUDA allows for simple and efficient distribution of these implementations across a multi-GPU NVIDIA DGX-2 server. We show that even a basic GPU implementation can outperform current results published on TPUs and that the optimized multi-GPU implementation can simulate very large lattices faster than custom FPGA solutions.

cs.DC↗

arXiv2018

Exascale Deep Learning for Climate Analytics

We extract pixel-level masks of extreme weather patterns using variants of Tiramisu and DeepLabv3+ neural networks. We describe improvements to the software frameworks, input pipeline, and the network training algorithms necessary to efficiently scale deep learning on the Piz Daint and Summit systems. The Tiramisu network scales to 5300 P100 GPUs with a sustained throughput of 21.0 PF/s and parallel efficiency of 79.0%. DeepLabv3+ scales up to 27360 V100 GPUs with a sustained throughput of 325.8 PF/s and a parallel efficiency of 90.7% in single precision. By taking advantage of the FP16 Tensor Cores, a half-precision version of the DeepLabv3+ network achieves a peak and sustained throughput of 1.13 EF/s and 999.0 PF/s respectively.

cs.DC↗