arXiv subjects

Wenyao Zhang

Publications and source records attributed to Wenyao Zhang.

31 records · Page 2Linked to original sources

arXiv2025

SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation

While spatial reasoning has made progress in object localization relationships, it often overlooks object orientation-a key factor in 6-DoF fine-grained manipulation. Traditional pose representations rely on pre-defined frames or templates, limiting generalization and semantic grounding. In this paper, we introduce the concept of semantic orientation, which defines object orientations using natural language in a reference-frame-free manner (e.g., the "plug-in" direction of a USB or the "handle" direction of a cup). To support this, we construct OrienText300K, a large-scale dataset of 3D objects annotated with semantic orientations, and develop PointSO, a general model for zero-shot semantic orientation prediction. By integrating semantic orientation into VLM agents, our SoFar framework enables 6-DoF spatial reasoning and generates robotic actions. Extensive experiments demonstrated the effectiveness and generalization of our SoFar, e.g., zero-shot 48.7% successful rate on Open6DOR and zero-shot 74.9% successful rate on SIMPLER-Env.

cs.RO↗

arXiv2025

Closed-Loop Unsupervised Representation Disentanglement with $β$-VAE Distillation and Diffusion Probabilistic Feedback

Representation disentanglement may help AI fundamentally understand the real world and thus benefit both discrimination and generation tasks. It currently has at least three unresolved core issues: (i) heavy reliance on label annotation and synthetic data -- causing poor generalization on natural scenarios; (ii) heuristic/hand-craft disentangling constraints make it hard to adaptively achieve an optimal training trade-off; (iii) lacking reasonable evaluation metric, especially for the real label-free data. To address these challenges, we propose a \textbf{C}losed-\textbf{L}oop unsupervised representation \textbf{Dis}entanglement approach dubbed \textbf{CL-Dis}. Specifically, we use diffusion-based autoencoder (Diff-AE) as a backbone while resorting to $β$-VAE as a co-pilot to extract semantically disentangled representations. The strong generation ability of diffusion model and the good disentanglement ability of VAE model are complementary. To strengthen disentangling, VAE-latent distillation and diffusion-wise feedback are interconnected in a closed-loop system for a further mutual promotion. Then, a self-supervised \textbf{Navigation} strategy is introduced to identify interpretable semantic directions in the disentangled latent space. Finally, a new metric based on content tracking is designed to evaluate the disentanglement effect. Experiments demonstrate the superiority of CL-Dis on applications like real image manipulation and visual analysis.

cs.CV↗

arXiv2025

DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge

Recent advances in vision-language-action (VLA) models have shown promise in integrating image generation with action prediction to improve generalization and reasoning in robot manipulation. However, existing methods are limited to challenging image-based forecasting, which suffers from redundant information and lacks comprehensive and critical world knowledge, including dynamic, spatial and semantic information. To address these limitations, we propose DreamVLA, a novel VLA framework that integrates comprehensive world knowledge forecasting to enable inverse dynamics modeling, thereby establishing a perception-prediction-action loop for manipulation tasks. Specifically, DreamVLA introduces a dynamic-region-guided world knowledge prediction, integrated with the spatial and semantic cues, which provide compact yet comprehensive representations for action planning. This design aligns with how humans interact with the world by first forming abstract multimodal reasoning chains before acting. To mitigate interference among the dynamic, spatial and semantic information during training, we adopt a block-wise structured attention mechanism that masks their mutual attention, preventing information leakage and keeping each representation clean and disentangled. Moreover, to model the conditional distribution over future actions, we employ a diffusion-based transformer that disentangles action representations from shared latent features. Extensive experiments on both real-world and simulation environments demonstrate that DreamVLA achieves 76.7% success rate on real robot tasks and 4.44 average length on the CALVIN ABC-D benchmarks.

cs.CV↗

arXiv2025

DexVLG: Dexterous Vision-Language-Grasp Model at Scale

As large models gain traction, vision-language-action (VLA) systems are enabling robots to tackle increasingly complex tasks. However, limited by the difficulty of data collection, progress has mainly focused on controlling simple gripper end-effectors. There is little research on functional grasping with large models for human-like dexterous hands. In this paper, we introduce DexVLG, a large Vision-Language-Grasp model for Dexterous grasp pose prediction aligned with language instructions using single-view RGBD input. To accomplish this, we generate a dataset of 170 million dexterous grasp poses mapped to semantic parts across 174,000 objects in simulation, paired with detailed part-level captions. This large-scale dataset, named DexGraspNet 3.0, is used to train a VLM and flow-matching-based pose head capable of producing instruction-aligned grasp poses for tabletop objects. To assess DexVLG's performance, we create benchmarks in physics-based simulations and conduct real-world experiments. Extensive testing demonstrates DexVLG's strong zero-shot generalization capabilities-achieving over 76% zero-shot execution success rate and state-of-the-art part-grasp accuracy in simulation-and successful part-aligned grasps on physical objects in real-world scenarios.

cs.CV↗

arXiv2024

Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion

Camera-based 3D semantic scene completion (SSC) is pivotal for predicting complicated 3D layouts with limited 2D image observations. The existing mainstream solutions generally leverage temporal information by roughly stacking history frames to supplement the current frame, such straightforward temporal modeling inevitably diminishes valid clues and increases learning difficulty. To address this problem, we present HTCL, a novel Hierarchical Temporal Context Learning paradigm for improving camera-based semantic scene completion. The primary innovation of this work involves decomposing temporal context learning into two hierarchical steps: (a) cross-frame affinity measurement and (b) affinity-based dynamic refinement. Firstly, to separate critical relevant context from redundant information, we introduce the pattern affinity with scale-aware isolation and multiple independent learners for fine-grained contextual correspondence modeling. Subsequently, to dynamically compensate for incomplete observations, we adaptively refine the feature sampling locations based on initially identified locations with high affinity and their neighboring relevant regions. Our method ranks $1^{st}$ on the SemanticKITTI benchmark and even surpasses LiDAR-based methods in terms of mIoU on the OpenOccupancy benchmark. Our code is available on https://github.com/Arlo0o/HTCL.

cs.CV↗

arXiv2024

Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation

There has been exciting progress in generating images from natural language or layout conditions. However, these methods struggle to faithfully reproduce complex scenes due to the insufficient modeling of multiple objects and their relationships. To address this issue, we leverage the scene graph, a powerful structured representation, for complex image generation. Different from the previous works that directly use scene graphs for generation, we employ the generative capabilities of variational autoencoders and diffusion models in a generalizable manner, compositing diverse disentangled visual clues from scene graphs. Specifically, we first propose a Semantics-Layout Variational AutoEncoder (SL-VAE) to jointly derive (layouts, semantics) from the input scene graph, which allows a more diverse and reasonable generation in a one-to-many mapping. We then develop a Compositional Masked Attention (CMA) integrated with a diffusion model, incorporating (layouts, semantics) with fine-grained attributes as generation guidance. To further achieve graph manipulation while keeping the visual content consistent, we introduce a Multi-Layered Sampler (MLS) for an "isolated" image editing effect. Extensive experiments demonstrate that our method outperforms recent competitors based on text, layout, or scene graph, in terms of generation rationality and controllability.

cs.CV↗

arXiv2023

Thermoelectric effect and temperature-gradient-driven electrokinetic flow of electrolyte solutions in charged nanocapillaries

A systematic theoretical study of thermoelectric effect and temperature-gradient-driven electrokinetic flow of electrolyte solutions in charged nanocapillaries is presented. The study is based on a semianalytical model developed by simultaneously solving the non-isothermal Poisson-Nernst-Planck-Navier-Stokes equations with the lubrication theory. Particularly, this paper clarifies the interplay and relative importance of the thermoelectric mechanisms due to (a) the convective transport of ions caused by the fluid flow, (b) the dependence of ion electrophoretic mobility on temperature, (c) the difference in the intrinsic Soret coefficients of cation and anion. Additionally, synergy conditions for the three thermoelectric mechanisms to fully cooperate are proposed for thermo-phobic/philic electrolytes. The temperature-gradient-driven electrokinetic flow is shown to be a nearly unidirectional flow whose axial velocity profiles vary with the axial location. Also, the flow can be regarded as a consequence of the counteraction or cooperation between a thermoelectric-field-driven electroosmotic flow and a thermo-osmotic flow driven by the osmotic pressure gradient and dielectric body force. Moreover, the Seebeck coefficient and the fluid average velocity are demonstrated to be affected by electrolyte-related parameters. The results are beneficial for understanding the temperature-gradient-driven electrokinetic transport in nanocapillaries and also serve as theoretical foundation for the design of low-grade waste heat recovery devices and thermoosmotic pumps.

physics.flu-dyn↗

arXiv2023

Simultaneous thermoosmotic and thermoelectric responses in nanoconfined electrolyte solutions: Effects of nanopore structures and membrane properties

Hypothesis: Nanofluidic systems provide an emerging and efficient platform for thermoelectric conversion and fluid pumping with low-grade heat energy. As a basis of their performance enhancement, the effects of the structures and properties of the nanofluidic systems on the thermoelectric response (TER) and the thermoosmotic response (TOR) are yet to be explored. Methods: The simultaneous TER and TOR of electrolyte solutions in nanofluidic membrane pores on which an axial temperature gradient is exerted are investigated numerically and semi-analytically. A semi-analytical model is developed with the consideration of finite membrane thermal conductivity and the reservoir/entrance effect. Findings: The increase in the access resistance due to the nanopore-reservoir interfaces accounts for the decrease of short circuit current at the low concentration regime. The decrease in the thermal conductivity ratio can enhance the TER and TOR. The maximum power density occurring at the nanopore radius twice the Debye length ranges from several to dozens of mW K$^{-2}$ m$^{-2}$ and is an order of magnitude higher than typical thermo-supercapacitors. The surface charge polarity can heavily affect the sign and magnitude of the short-circuit current, the Seebeck coefficient, and the open-circuit thermoosmotic coefficient, but has less effect on the short-circuit thermoosmotic coefficient. Furthermore, the membrane thickness makes different impacts on TER and TOR for zero and finite membrane thermal conductivity.

cond-mat.soft↗

arXiv2023

Ion steric effect induces giant enhancement of thermoelectric conversion in electrolyte-filled nanochannels

Ionic thermoelectricity in nanochannels has received increasing attention because of its advantages such as high Seebeck coefficient and low cost. However, most studies have focused on dilute simple electrolytes that neglect the effects of finite ion sizes and short-range electrostatic correlation. Here, we reveal a new thermoelectric mechanism arising from the coupling of ion steric effect due to finite ion sizes and ion thermodiffusion in electric double layers, using both theoretical and numerical methods. We show that this mechanism can significantly enhance the thermoelectric response in nanoconfined electrolytes, depending on the properties of electrolytes and nanochannels. Compared to the previously known mechanisms, the new mechanism can increase the Seebeck coefficient by 100\% or even one order of magnitude enhancement under optimal conditions. Moreover, we demonstrate that the short-range electrostatic correlation can help preserve the Seebeck coefficient enhancement in weaker confinement or in more concentrated electrolytes.

cond-mat.soft↗

arXiv2023

[CLS] Token is All You Need for Zero-Shot Semantic Segmentation

In this paper, we propose an embarrassingly simple yet highly effective zero-shot semantic segmentation (ZS3) method, based on the pre-trained vision-language model CLIP. First, our study provides a couple of key discoveries: (i) the global tokens (a.k.a [CLS] tokens in Transformer) of the text branch in CLIP provide a powerful representation of semantic information and (ii) these text-side [CLS] tokens can be regarded as category priors to guide CLIP visual encoder pay more attention on the corresponding region of interest. Based on that, we build upon the CLIP model as a backbone which we extend with a One-Way [CLS] token navigation from text to the visual branch that enables zero-shot dense prediction, dubbed \textbf{ClsCLIP}. Specifically, we use the [CLS] token output from the text branch, as an auxiliary semantic prompt, to replace the [CLS] token in shallow layers of the ViT-based visual encoder. This one-way navigation embeds such global category prior earlier and thus promotes semantic segmentation. Furthermore, to better segment tiny objects in ZS3, we further enhance ClsCLIP with a local zoom-in strategy, which employs a region proposal pre-processing and we get ClsCLIP+. Extensive experiments demonstrate that our proposed ZS3 method achieves a SOTA performance, and it is even comparable with those few-shot semantic segmentation methods.

cs.CV↗

arXiv2022

Predict the Rover Mobility over Soft Terrain using Articulated Wheeled Bevameter

Robot mobility is critical for mission success, especially in soft or deformable terrains, where the complex wheel-soil interaction mechanics often leads to excessive wheel slip and sinkage, causing the eventual mission failure. To improve the success rate, online mobility prediction using vision, infrared imaging, or model-based stochastic methods have been used in the literature. This paper proposes an on-board mobility prediction approach using an articulated wheeled bevameter that consists of a force-controlled arm and an instrumented bevameter (with force and vision sensors) as its end-effector. The proposed bevameter, which emulates the traditional terramechanics tests such as pressure-sinkage and shear experiments, can measure contact parameters ahead of the rover's body in real-time, and predict the slip and sinkage of supporting wheels over the probed region. Based on the predicted mobility, the rover can select a safer path in order to avoid dangerous regions such as those covered with quicksand. Compared to the literature, our proposed method can avoid the complicated terramechanics modeling and time-consuming stochastic prediction; it can also mitigate the inaccuracy issues arising in non-contact vision-based methods. We also conduct multiple experiments to validate the proposed approach.

cs.RO↗

arXiv2020

Prediction of Ternary Fluorooxoborates with Coplanar Triangle Units [BOxF3-x]x- From First-Principles

Ten new ternary fluorooxoborate structures were obtained from first-principles prediction. Coplanar aligned triangle structure units [BO2F]2- and [BOF2]- like [BO3]3- in borates were found from the computational simulation. We identified new covalent coordination patterns of the F atom connected with the B atoms which are located in the bridging site, -B--F--B-. Besides, one molecular crystal with [B4O4F4] molecular unit was attached.

cond-mat.mtrl-sci↗

arXiv2019

Temperature-gradient-induced electrokinetic flow and thermoelectricity of electrolyte solutions in a capillaries

A systematic theoretical study of temperature-gradient-induced electrokinetic flow and thermoelectric potential of electrolyte solutions in a micro-/nanocapillary is presented. The study is based on a semi-analytical model developed by simultaneously solving the energy equation and the Poisson-Nernst-Planck/Navier-Stokes equations with the lubrication theory. The semi-analytical model is shown to be mainly governed by eight parameters, including two temperature-related parameters (temperature and its gradient), two electrokinetic parameters ($ζ$ potential and the ratio of capillary radius to the Debye length $κ_0a$) and four physical properties of cation and anion (i.e. Soret coefficient difference $ΔS_T$, average Soret coefficient $S_T$, normalized difference in diffusivities $χ$ and intrinsic Peclet number $λ$). It is found that the thermoelectric field is induced by three effects, which are respectively due to (1) the difference in the Soret coefficients of cation and anion; (2) the selective ion diffusion resulting from the temperature-modified Boltzmann distribution of ions; (3) the advective transport of ions caused by the fluid flow. The first thermoelectric effect prevails for lower $ζ$ potentials or large $κ_0a$, while the second is dominant for higher $ζ$ potentials with very small $κ_0a$. The first two thermoelectric effects can cooperate or counteract depending on the sign of $ζΔS_T$. Finally, the temperature-gradient-induced electrokinetic flow is found to be a superposition of an electroosmotic flow component due to the thermoelectric field and a thermoosmotic flow component due to the combined effects of osmotic pressure and dielectric body force. These two flow components may cooperate or counteract depending on values of $ζ$ and $κ_0a$.

physics.flu-dyn↗