Searcharxiv⌕ Search

arXiv subjects

Pascal Fua

Publications and source records attributed to Pascal Fua.

At least 19 recordsLinked to original sources
arXiv2026

Revisiting Multi-View Stereo: A Sequence-to-Sequence Formulation

Computing accurate geometry from multi-view images is a fundamental problem in computer vision. Recent feed-forward (FF) models jointly estimate 3D geometry and camera parameters, but they typically suffer from geometry distortion caused by reconstruction ambiguity, even when ground-truth camera parameters are supplied. In this paper, we study the multi-view stereo (MVS) problem with known camera parameters and propose a novel approach that bridges conventional MVS and FF methods. Rather than casting MVS as a sequence-to-one mapping that predicts depth only for a single reference view, we reformulate it as a sequence-to-sequence task, akin to FF models, that jointly predicts geometry for all input views. We introduce a global transformer-based architecture with two components that explicitly exploit camera-induced priors: ray-map embeddings that inject camera parameters into image patch tokens, making the transformer camera-aware, and a unified global cost volume that replaces conventional per-view cost volumes to jointly capture 3D structure across all views. Extensive experiments on multiple public benchmarks show our approach achieves state-of-the-art performance, surpassing both MVS and FF reconstruction baselines.

cs.CV↗
arXiv2026

PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video

Human motion, environmental contacts, and interaction forces are governed by common physical laws, yet existing approaches typically separate visual pose reconstruction from contact and force estimation. This separation limits joint reasoning and can propagate errors between stages. We introduce PACT, an end-to-end model that jointly learns to estimate human pose, contacts and contact forces from monocular video. Our approach augments a human reconstruction foundation model with learnable contact-force tokens and a temporal transformer that integrates visual features with world-space motion. Joint prediction heads refine human poses and estimate contacts and forces, while physics-based supervision encourages consistency between the reconstructed motion and interaction forces. To address the scarcity of force annotations, we develop a data annotation pipeline that combines contact labeling with physics-based motion and force optimization, producing training supervision from synthetic and real-world videos. We also introduce a real-world climbing benchmark ForceWall with climbing videos and corresponding ground-truth contact forces obtained from the force sensors. Experiments demonstrate state-of-the-art contact and force estimation, outperforming staged reconstruction approaches and generalizing to interactions beyond the training distribution. These results support end-to-end joint learning as an effective approach to recovering human motion and physical interactions from video.

cs.CV↗
arXiv2026

WindINR: Latent-State INR for Fast Local Wind Query and Correction in Complex Terrain

Many decisions in complex terrain require fast wind estimates at specific locations and heights rather than dense forecasts on fixed grids. We present WindINR, a latent-state implicit neural representation framework for continuous high-resolution local wind queries and correction using sparse observations. A latent-conditioned decoder maps terrain descriptors, a low-resolution background field, and continuous query coordinates to local wind estimates. WindINR separates reusable representation learning from sample-specific latent correction. During training, a privileged encoder infers reference latent states from high-resolution supervision, while a deployable predictor estimates initial states from inference-time inputs. Their discrepancies define a dataset-adaptive Gaussian prior over latent corrections. At inference time, network weights remain fixed, and only the latent state is updated through a regularized objective incorporating sparse observations and their uncertainty. We train WindINR on the large-scale FuXi-CFD dataset and evaluate it using controlled observing system simulation experiments, simulated helicopter trajectories, and real UAV measurements over Tromsø. Across these settings, sparse-observation assimilation consistently improves local wind predictions, with WindINR outperforming several baselines, including WindSeer. Updating only a low-dimensional latent state yields a 14-fold speedup over full-network fine-tuning while preserving continuous queries at arbitrary spatial coordinates. WindINR thus provides an efficient interface between kilometer-scale background products, sparse local observations, and local wind queries in complex terrain.

cs.AI↗
arXiv2026

Neural-Surrogate Co-Design of Aerodynamic Shape and Control

Control co-design jointly optimizes a robot's physical design and its controller. Existing techniques are limited in the number of model parameters and the complexity of the physical models they can handle. In this paper, we propose a gradient-based bilevel formulation that replaces the dynamics models of earlier techniques with a differentiable neural surrogate and differentiates through the trajectory optimizer using a structure-exploiting approach. We validate the framework on fixed-wing gliders, using NeuralFoil as the aerodynamic surrogate. The tasks are maximum-distance gliding, minimum-distance landing, and perching. Experiments on 2D airfoils and 3D wings show that our method runs about five times faster than a state-of-the-art evolutionary baseline. It also outperforms pipelines that optimize aerodynamic proxy metrics.

cs.RO↗
arXiv2026

PhysMoDPO: Physically-Plausible Humanoid Motion with Preference Optimization

Recent progress in text-conditioned human motion generation has been largely driven by diffusion models trained on large-scale human motion data. Building on this progress, recent methods attempt to transfer such models for character animation and real robot control by applying a Whole-Body Controller (WBC) that converts diffusion-generated motions into executable trajectories. While WBC trajectories become compliant with physics, they may expose substantial deviations from original motion. To address this issue, we here propose PhysMoDPO, a Direct Preference Optimization framework. Unlike prior work that relies on hand-crafted physics-aware heuristics such as foot-sliding penalties, we integrate WBC into our training pipeline and optimize diffusion model such that the output of WBC becomes compliant both with physics and original text instructions. To train PhysMoDPO we deploy physics-based and task-specific rewards and use them to assign preference to synthesized trajectories. Our extensive experiments on text-to-motion and spatial control tasks demonstrate consistent improvements of PhysMoDPO in both physical realism and task-related metrics on simulated robots. Moreover, we demonstrate that PhysMoDPO results in significant improvements when applied to zero-shot motion transfer in simulation and for real-world deployment on a G1 humanoid robot.

cs.LG↗
arXiv2026

StyleFields: Multi-Scale AdaIN-Modulated Implicit SDFs for Coarse-to-Fine 3D Shape Reconstruction and Editing

We introduce StyleFields, a DeepSDF-based architecture for high-fidelity 3D reconstruction that enables controllable geometric style mixing: the coarse structure of one object can be combined with the fine-scale details of another. The core idea is depth-aware modulation: instead of a single global code, we inject latents via multi-level Adaptive Instance Normalization at several decoder depths, and supervise matching auxiliary heads with a coarse-to-fine schedule while gradually growing network depth. This aligns early layers with global shape and later layers with high-frequency detail, achieving content-style decoupling without part labels or adversarial training. StyleFields delivers faithful reconstructions, convincing cross-instance hybrids, and consistent gains in ablations over injection depth and supervision granularity. We further demonstrate a practical application in automotive aerodynamics: a learned surrogate drag predictor serves as a differentiable objective to optimize reconstructed cars, allowing targeted edits of global form or surface details by freezing the complementary latent stream. StyleFields offers a simple, effective recipe for controllable implicit reconstruction and downstream performance-driven design.

cs.CV↗
arXiv2026

Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning

With massively parallel simulation, on-policy Reinforcement Learning methods such as PPO have become standard in many domains. However, learning from scratch is sample-inefficient and fails to exploit the potential existence of a suboptimal expert, such as a heuristic, a model-based controller, or a policy trained on a related task. Such an expert is often available and can guide early training, but its sub-optimality limits final performance. The challenge then becomes balancing expert guidance against learning from rewards. Existing methods set the expert's influence through a blending weight, a schedule, or an evaluation-driven curriculum. Alternatively, they adapt it with additional learned components such as critics over expert actions or auxiliary agents. However, none optimizes it using the same on-policy objective as the policy itself. We propose a method in which the learner and the expert alternate control within each training episode, and the expert's share of control is a single learnable parameter optimized jointly with the policy. The learner benefits from the expert early in training, but its share of control declines as the learner becomes more competent, until eventually vanishing completely. This leaves the learner acting alone and better than the suboptimal expert. We evaluate our method on 34 tasks across two benchmarks, spanning discrete and continuous action spaces, using both learned and model-based experts. Our method improves sample efficiency over guided and unguided baselines while requiring minimal hyperparameter variation. The expert's share decays to zero as the learner improves, vanishing when the expert is no longer useful.

cs.LG↗
arXiv2026

AnchorGen: Anchored Optimization for Customizable Generative 3D Design

Engineering design often starts from a 2D sketch that fixes style and proportions, yet the subsequent 3D shape optimization relies on learned generative priors to keep the geometry valid. However, these priors are agnostic to the sketch: while they admit a valid design by correcting a drifted proposal back to its training distribution, they often correct it towards the high-density region, ignoring the specified design. We introduce \emph{AnchorGen}, a rectified-flow framework trained unconditionally on the concatenated shape and sketch latents of paired data. The learned manifold represents the joint distribution of shape-sketch pairs, so constraining the sketch component restricts the iterate to the sub-manifold of shapes consistent with a target style. Since training employs no conditioning signal, the constraint is imposed at inference: gradient descent optimizes the shape latent to minimize a differentiable drag surrogate, while constraining the sketch latent to remain close to the target sketch via a token-wise cosine penalty. A single model thereby supports design-preserving optimization, dimensionally explicit design edits, and sketch-only synthesis.

cs.CV↗
arXiv2026

Generative Refinement for Low-Budget Black-Box Optimization

Black-box optimization is a fundamental tool in science and engineering for optimizing objectives when gradient information is unavailable. It becomes especially difficult when the objective function is expensive to evaluate, limiting the evaluation budget to a few tens or hundreds of queries, and when good solutions occupy complex, low-measure regions of the search space. Generative models can supply useful structural priors in such settings, but existing generative BBO approaches bring significant evaluation cost. We identify three design principles for generative optimization under such low-budget conditions: avoid objective learning, optimize in candidate space, and make every evaluation count. Together, these principles motivate separating structural modeling from objective-driven search. We instantiate them in SPARROW, a simple sequential optimizer that maintains a persistent, ranked archive of evaluated candidates and uses a fixed, unconditional generative sampler solely as a corruption-refinement operator. SPARROW requires only access to the sampler's corruption and refinement processes, and never needs to evaluate the objective to train or guide it. Across three complementary settings, probing thin feasible geometry, disconnected high-performing regions, and failure-prone evaluations, SPARROW outperforms classical and generative baselines under strict evaluation budgets. These results demonstrate that separating structural priors from objective-driven search can be effective when evaluations are scarce and the search geometry is challenging.

cs.LG↗
arXiv2026

Generalizable Neural Reconstruction of High-Fidelity Surfaces via Sparse Volumetric Representations

Neural implicit representations have recently achieved impressive results in novel view synthesis and multi-view 3D reconstruction, yet both NeRF- and Gaussian Splatting-based methods require per-scene optimization, which makes them inefficient. Generalizable Neural Surface Reconstruction (GNSR) methods have been proposed to remove this need by learning feature representations directly predicted from input images. However, their typical reliance on dense feature volumes severely limits achievable resolution and fidelity due to prohibitive memory costs. We introduce Sparse Volumetric Reconstruction (SVRecon), a new GNSR framework that unlocks high-resolution, memory-efficient reconstruction through learned occupancy-driven sparsity, in a more effective way than earlier approaches to introducing sparsity in GNSRs. Our approach uses a nested two-stage architecture: (1) an occupancy prediction network that identifies surface-containing voxels, and (2) a high-resolution sparse volume rendering framework defined only within these occupied regions, together with specialized sparsified algorithms for ray sampling, feature aggregation, and querying. This design enables fine-grained surface reconstruction while avoiding the heavy memory footprint of dense grids. SVRecon operates at resolutions up to $512^3$ on standard 32GB hardware---substantially higher than prior generalizable methods---and delivers smoother and more precise reconstructions across diverse datasets, particularly in sparse-view settings.

cs.CV↗
arXiv2026

S2MDF: A Plug-And-Play Layer for Intersection-Free Multi-Object Signed Distance Fields

Compositional implicit surface representations model scenes as collections of objects, each encoded by a Signed Distance Field (SDF). A fundamental limitation of this approach is that multiple SDFs can produce geometries that interpenetrate, violating physical plausibility. Existing mitigation strategies rely on soft penalty terms that reduce but do not eliminate intersections, and require careful loss weighting. To truly prevent interpenetration, we propose a hard constraint on vector-valued SDFs and introduce S2MDF, a lightweight plug-and-play module that enforces the constraint on any object-compositional SDF representation without architectural modifications. It introduces negligible computational overhead and is compatible with linearly-interpolated standard meshing algorithms such as Marching Cubes. It can be applied during training or as a post-processing step. Experiments on multiple state-of-the-art compositional methods show that S2MDF reduces intersections to numerical precision while preserving reconstruction quality, outperforming existing mitigation strategies.

cs.CV↗
arXiv2026

Aerodynamic Shape Design Space Exploration with Deep Latent Diffusion Model

We propose DiffGeo, a latent space diffusion-based generative framework for aerodynamic design space exploration under extreme data scarcity. DiffGeo combines a learned latent space model for automatic shape parameterization, with a diffusion sampler to directly generate novel, geometry-valid and controllable designs. We validate the approach on a series of case studies: (i) a 2D airfoil generation benchmark, where DiffGeo's latent diffusion model is compared against GAN- and VAE-based baselines in terms of sample quality, diversity and constraint adherence under limited data; (ii) integration into a surrogate-based optimization pipeline, where DiffGeo's conditional sampling produces task-informed airfoil data that improve both surrogate modeling and optimization performance; and (iii) extension to 3D turbomachinery blade prototyping, where DiffGeo generates realistic and high-performance blade geometries from a small set of reference designs. Throughout these investigations, DiffGeo achieves high-quality and diverse shape generation with at least an order of magnitude less data than alternatives, decouples geometry representation from design targets for flexible reuse, and seamlessly incorporates complex design constraints via energy-based conditioning. These capabilities demonstrate DiffGeo's potential to enhance early-stage design by automating design space exploration--improving efficiency, expanding design diversity and embedding engineering knowledge through controllable guidance.

cs.CE↗
arXiv2026

Spatio-Temporal Garment Reconstruction Using Diffusion Mapping via Pattern Coordinates

Reconstructing 3D clothed humans from monocular images and videos is a fundamental problem with applications in virtual try-on, avatar creation, and mixed reality. Despite significant progress in human body recovery, accurately reconstructing garment geometry, particularly for loose-fitting clothing, remains an open challenge. We propose a unified framework for high-fidelity 3D garment reconstruction from both single images and video sequences. Our approach combines Implicit Sewing Patterns (ISP) with a generative diffusion model to learn expressive garment shape priors in 2D UV space. Leveraging these priors, we introduce a mapping model that establishes correspondences between image pixels, UV pattern coordinates, and 3D geometry, enabling accurate and detailed garment reconstruction from single images. We further extend this formulation to dynamic reconstruction by introducing a spatio-temporal diffusion scheme with test-time guidance to enforce long-range temporal consistency. We also develop analytic projection-based constraints that preserve image-aligned geometry in visible regions while enforcing coherent completion in occluded areas over time. Although trained exclusively on synthetically simulated cloth data, our method generalizes well to real-world imagery and consistently outperforms existing approaches on both tight- and loose-fitting garments. The reconstructed garments preserve fine geometric detail while exhibiting realistic dynamic motion, supporting downstream applications such as texture editing, garment retargeting, and animation.

cs.CV↗
arXiv2026

AvatarDynamizer: From Static to Dynamic Human Avatars via Generative Dynamic Textures

For full-body avatars, modeling surface dynamics is crucial for overcoming the uncanny valley and achieving perceptual realism. Person-agnostic methods recover static 3D avatars from monocular images, videos, or text prompts, but their skeleton-driven animations lack realistic surface dynamics such as clothing wrinkles. In contrast, person-specific methods achieve high-quality rendering and realistic dynamics, but require expensive multi-view captures for each individual. Recent generalizable dynamic avatar methods struggle to embed surface dynamics, leading to either limited multi-view consistency or dynamic expressiveness. To this end, we propose AvatarDynamizer, a generative method that transforms an off-the-shelf static 3D avatar into a controllable, realistic, and multi-view-consistent 4D avatar. We introduce a novel texture-space surface-dynamics embedding and formulate avatar dynamics modeling as conditional texture generation. Our encoder--decoder representation embeds pose-dependent dynamics into dynamic texture maps, enabling compatibility with pre-trained video diffusion models while decoding them into 3D Gaussians for multi-view consistent rendering. Since existing datasets are limited in scale, sequence length, or motion diversity, we collect a large-scale multi-view dataset with long sequences covering diverse skeletal motions and surface dynamics. Experiments show that our method effectively animates static avatars with faithful surface dynamics and outperforms competing generalizable methods in visual fidelity, especially under limited dynamic training data.

cs.CV↗
arXiv2026

Flow-Corrected Shape Optimization: Taming Manifold Drift in High-Dimensional 3D Models

Optimizing 3D shapes within the latent spaces of deep generative models is fundamental to computer assisted engineering, yet remains prone to a critical failure mode we term manifold drift: the tendency of gradient-based optimization to move latent vectors away from the manifold of valid shapes. This problem is exacerbated in state-of-the-art 3D shape generative models that operate in increasingly high-dimensional latent spaces where valid shapes occupy a vanishingly small fraction of the full space. Existing mitigation strategies, including latent regularization and flow-matching approaches, either sacrifice expressiveness, demand a difficult trade-off between objective guidance and generative fidelity that remains prone to manifold drift, or are computationally infeasible to scale to modern, large-capacity 3D shape models. We introduce a novel optimizer-corrector framework that alternates between gradient steps for objective minimization and guided flow matching to drive the latent state back to the valid shape manifold. By decoupling objective minimization from flow-based correction, optimizing freely and correcting strictly, this alternating design avoids inherent trade-offs, preserving geometric validity without sacrificing expressiveness while remaining computationally feasible on modern 3D shape models. We demonstrate its effectiveness across generative priors of varying complexity, from simple vector latent spaces to large-scale architectures across a variety of downstream optimization tasks, including aerodynamic drag reduction and object compliance optimization.

cs.CV↗
arXiv2026

End-to-End 4D Heart Mesh Recovery Across Full-Stack and Sparse Cardiac MRI

Reconstructing cardiac motion from CMR sequences is critical for diagnosis, prognosis, and intervention. Existing methods rely on complete CMR stacks to infer full heart motion, limiting their applicability during intervention when only sparse observations are available. We present TetHeart, the first end-to-end framework for unified 4D heart mesh recovery from both offline full-stack and intra-procedural sparse-slice observations. Our method leverages deformable tetrahedra to capture shape and motion in a coherent space shared across cardiac structures. Before a procedure, it initializes detailed, patient-specific heart meshes from high-quality full stacks, which can then be updated using whatever slices can be obtained in real-time, down to a single one during the procedure. TetHeart incorporates several key innovations: (i) an attentive slice-adaptive 2D-3D feature assembly mechanism that integrates information from arbitrary numbers of slices at any position; (ii) a distillation strategy to ensure accurate reconstruction under extreme sparsity; and (iii) a weakly supervised motion learning scheme requiring annotations only at keyframes, such as the end-diastolic and end-systolic phases. Trained and validated on three large public datasets and evaluated zero-shot on additional private interventional and public datasets without retraining, TetHeart achieves state-of-the-art accuracy and strong generalization in both pre- and intra-procedural settings. Code and dataset is available at https://github.com/Scalsol/TetHeart.

cs.CV↗
arXiv2026

VecHeart: Holistic Four-Chamber Cardiac Anatomy Modeling via Hybrid VecSets

Accurate cardiac anatomy modeling requires the model to be able to handle intricate interrelations among structures. In this paper, we propose VecHeart, a unified framework for holistic reconstruction and generation of four-chamber cardiac structures. To overcome the limitations of current feed-forward implicit methods, specifically their restriction to single-object modeling and their neglect of inter-part correlations, we introduce Hybrid Part Transformer, which leverages part-specific learnable queries and interleaved attention to capture complex inter-chamber dependencies. Furthermore, we propose Anatomical Completion Masking and Modality Alignment strategies, enabling the model to infer complete four-chamber structures from partial, sparse, or noisy observations, even when certain anatomical parts are entirely missing. VecHeart also seamlessly extends to 3D+t dynamic mesh sequence generation, demonstrating exceptional versatility. Experiments show that our method achieves state-of-the-art performance, maintaining high-fidelity reconstruction across diverse challenging scenarios. Code is available at https://github.com/Scalsol/VecHeart.

cs.CV↗
arXiv2026

MirrorCheck: Efficient Adversarial Defense for Vision-Language Models

Vision-Language Models (VLMs) are increasingly susceptible to sophisticated adversarial attacks, including adaptive strategies specifically designed to bypass existing defenses. To address this vulnerability, we propose MirrorCheck, a robust and model-agnostic detection framework that operates effectively in both unimodal and multimodal settings. MirrorCheck leverages Text-to-Image (T2I) models to regenerate visual content from captions produced by the target model and assesses semantic consistency by comparing feature-space embeddings between the original and synthesized images. To enhance robustness against adaptive attacks, MirrorCheck introduces a stochastic defense strategy that randomly selects T2I generators and image encoders from a diverse model zoo. Additionally, we incorporate a novel One-Time-Use (OTU) perturbation applied to the selected encoder embeddings, regulated by a scaling factor, which decreases the effectiveness of adaptive attacks. Extensive experiments across multiple threat scenarios demonstrate that MirrorCheck consistently outperforms baseline methods, and maintains its utility even under strong adaptive adversarial conditions.

cs.CV↗