arXiv subjects

Zhiwei Jia

Publications and source records attributed to Zhiwei Jia.

At least 19 recordsLinked to original sources

arXiv2026

Wide-field mid-infrared cavity-enhanced upconversion imaging

Mid-infrared (MIR) spectral imaging enables precise target identification and analysis by capturing rich chemical fingerprints, which calls for high-sensitivity broadband MIR imagers at room temperature. Here, we devise and implement a continuous-wave pumping MIR upconversion imaging system based on external-cavity enhancement, which favors a large field of view, a low cavity loss, and a high spectral resolution. The involved optical cavity is constructed in an integrated fashion by utilizing one crystal facet as a cavity mirror, which allows a 43-fold power enhancement for the single-longitudinal-mode pump at 1064 nm. In combination with the chirped-poled crystal design, high-fidelity and wide-field spectral imaging mapping is permitted to facilitate an acceptance angle up to 28.5$^\circ$ over a spectral coverage of 2.5--5 $μ$m. Moreover, a thermal locking approach is used to stabilize the cavity at the high-power operation, eliminating active feedback and ensuring long-term stability. A proof-of-principle demonstration is presented to showcase real-time observation of CO$_{2}$ gas injection dynamics. The implemented MIR upconversion imager features wide-field operation, high detection sensitivity, and compact footprint, which would benefit subsequent applications including environment monitoring, gas leakage inspection, and medical diagnostics.

physics.optics↗

arXiv2025

Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward

Recent research has shown that fine-tuning diffusion models (DMs) with arbitrary rewards, including non-differentiable ones, is feasible with reinforcement learning (RL) techniques, enabling flexible model alignment. However, applying existing RL methods to step-distilled DMs is challenging for ultra-fast ($\le2$-step) image generation. Our analysis suggests several limitations of policy-based RL methods such as PPO or DPO toward this goal. Based on the insights, we propose fine-tuning DMs with learned differentiable surrogate rewards. Our method, named LaSRO, learns surrogate reward models in the latent space of SDXL to convert arbitrary rewards into differentiable ones for effective reward gradient guidance. LaSRO leverages pre-trained latent DMs for reward modeling and tailors reward optimization for $\le2$-step image generation with efficient off-policy exploration. LaSRO is effective and stable for improving ultra-fast image generation with different reward objectives, outperforming popular RL methods including DDPO and Diffusion-DPO. We further show LaSRO's connection to value-based RL, providing theoretical insights. See our webpage \href{https://sites.google.com/view/lasro}{here}.

cs.LG↗

arXiv2025

The uniqueness of minimal maps into Cartan-Hadamard manifolds via the squared singular values

In this paper, we give a uniqueness theorem for the Dirichlet problem of minimal maps into general Riemannian manifolds with non-positive sectional curvature, improving Theorem 5.2 of Lee-Ooi-Tsui's paper published in J. Geom. Anal.. The proof of this theorem is based on the convexity of several functions in terms of squared singular values along the geodesic homotopy of two given minimal maps.

math.DG↗

arXiv2024

Chain-of-Thought Predictive Control

We study generalizable policy learning from demonstrations for complex low-level control (e.g., contact-rich object manipulations). We propose a novel hierarchical imitation learning method that utilizes sub-optimal demos. Firstly, we propose an observation space-agnostic approach that efficiently discovers the multi-step subskill decomposition of the demos in an unsupervised manner. By grouping temporarily close and functionally similar actions into subskill-level demo segments, the observations at the segment boundaries constitute a chain of planning steps for the task, which we refer to as the chain-of-thought (CoT). Next, we propose a Transformer-based design that effectively learns to predict the CoT as the subskill-level guidance. We couple action and subskill predictions via learnable prompt tokens and a hybrid masking strategy, which enable dynamically updated guidance at test time and improve feature representation of the trajectory for generalizable policy learning. Our method, Chain-of-Thought Predictive Control (CoTPC), consistently surpasses existing strong baselines on challenging manipulation tasks with sub-optimal demos.

cs.LG↗

arXiv2023

Physical-layer key distribution using synchronous complex dynamics of DBR semiconductor lasers

Common-signal-induced synchronization of semiconductor lasers with optical feedback inspired a promising physical key distribution with information-theoretic security and potential in high rate. A significant challenge is the requirement to shorten the synchronization recovery time for increasing key rate without sacrificing operation parameter space for security. Here, open-loop synchronization of wavelength-tunable multi-section distributed Bragg reflector (DBR) lasers is proposed as a solution for physical-layer key distribution. Experiments show that the synchronization is sensitive to two operation parameters, i.e., currents of grating section and phase section. Furthermore, fast wavelength-shift keying synchronization can be achieved by direct modulation on one of the two currents. The synchronization recovery time is shortened by one order of magnitude compared to close-loop synchronization. An experimental implementation is demonstrated with a final key rate of 5.98 Mbit/s over 160 km optical fiber distance. It is thus believed that fast-tunable multi-section semiconductor lasers opens a new avenue of high-rate physical-layer key distribution using laser synchronization.

physics.optics↗

arXiv2023

MetaCLUE: Towards Comprehensive Visual Metaphors Research

Creativity is an indispensable part of human cognition and also an inherent part of how we make sense of the world. Metaphorical abstraction is fundamental in communicating creative ideas through nuanced relationships between abstract concepts such as feelings. While computer vision benchmarks and approaches predominantly focus on understanding and generating literal interpretations of images, metaphorical comprehension of images remains relatively unexplored. Towards this goal, we introduce MetaCLUE, a set of vision tasks on visual metaphor. We also collect high-quality and rich metaphor annotations (abstract objects, concepts, relationships along with their corresponding object boxes) as there do not exist any datasets that facilitate the evaluation of these tasks. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations, highlighting strengths and weaknesses of current approaches in visual metaphor Classification, Localization, Understanding (retrieval, question answering, captioning) and gEneration (text-to-image synthesis) tasks. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities.

cs.CV↗

arXiv2023

KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models

Image ad understanding is a crucial task with wide real-world applications. Although highly challenging with the involvement of diverse atypical scenes, real-world entities, and reasoning over scene-texts, how to interpret image ads is relatively under-explored, especially in the era of foundational vision-language models (VLMs) featuring impressive generalizability and adaptability. In this paper, we perform the first empirical study of image ad understanding through the lens of pre-trained VLMs. We benchmark and reveal practical challenges in adapting these VLMs to image ad understanding. We propose a simple feature adaptation strategy to effectively fuse multimodal information for image ads and further empower it with knowledge of real-world entities. We hope our study draws more attention to image ad understanding which is broadly relevant to the advertising industry.

cs.CV↗

arXiv2022

Learning to Act with Affordance-Aware Multimodal Neural SLAM

Recent years have witnessed an emerging paradigm shift toward embodied artificial intelligence, in which an agent must learn to solve challenging tasks by interacting with its environment. There are several challenges in solving embodied multimodal tasks, including long-horizon planning, vision-and-language grounding, and efficient exploration. We focus on a critical bottleneck, namely the performance of planning and navigation. To tackle this challenge, we propose a Neural SLAM approach that, for the first time, utilizes several modalities for exploration, predicts an affordance-aware semantic map, and plans over it at the same time. This significantly improves exploration efficiency, leads to robust long-horizon planning, and enables effective vision-and-language grounding. With the proposed Affordance-aware Multimodal Neural SLAM (AMSLAM) approach, we obtain more than 40% improvement over prior published work on the ALFRED benchmark and set a new state-of-the-art generalization performance at a success rate of 23.48% on the test unseen scenes.

cs.RO↗

arXiv2022

Improving Policy Optimization with Generalist-Specialist Learning

Generalization in deep reinforcement learning over unseen environment variations usually requires policy learning over a large set of diverse training variations. We empirically observe that an agent trained on many variations (a generalist) tends to learn faster at the beginning, yet its performance plateaus at a less optimal level for a long time. In contrast, an agent trained only on a few variations (a specialist) can often achieve high returns under a limited computational budget. To have the best of both worlds, we propose a novel generalist-specialist training framework. Specifically, we first train a generalist on all environment variations; when it fails to improve, we launch a large population of specialists with weights cloned from the generalist, each trained to master a selected small subset of variations. We finally resume the training of the generalist with auxiliary rewards induced by demonstrations of all specialists. In particular, we investigate the timing to start specialist training and compare strategies to learn generalists with assistance from specialists. We show that this framework pushes the envelope of policy learning on several challenging and popular benchmarks including Procgen, Meta-World and ManiSkill.

cs.LG↗

arXiv2022

Fast all-optical random number generator

We propose a simple and all-optical method for fast random number generation based on the laser mode hopping. Through periodically restarting a two-mode laser operating in the bistable state, a random number stream can be generated due to the spontaneous emission noise. To validate the feasibility of this method, we perform a theoretical simulation using the common vertical-cavity surface-emitting laser (VCSEL) with two polarization modes. Numerical results demonstrate that fast 2.5 Gb/s random number streams can be continuously obtained with verified randomness. This scheme provide a fully monolithic solution for random number generator, due to its simple and all-optical structure.

physics.optics↗

arXiv2021

TRIG: Transformer-Based Text Recognizer with Initial Embedding Guidance

Scene text recognition (STR) is an important bridge between images and text, attracting abundant research attention. While convolutional neural networks (CNNS) have achieved remarkable progress in this task, most of the existing works need an extra module (context modeling module) to help CNN to capture global dependencies to solve the inductive bias and strengthen the relationship between text features. Recently, the transformer has been proposed as a promising network for global context modeling by self-attention mechanism, but one of the main shortcomings, when applied to recognition, is the efficiency. We propose a 1-D split to address the challenges of complexity and replace the CNN with the transformer encoder to reduce the need for a context modeling module. Furthermore, recent methods use a frozen initial embedding to guide the decoder to decode the features to text, leading to a loss of accuracy. We propose to use a learnable initial embedding learned from the transformer encoder to make it adaptive to different input images. Above all, we introduce a novel architecture for text recognition, named TRansformer-based text recognizer with Initial embedding Guidance (TRIG), composed of three stages (transformation, feature extraction, and prediction). Extensive experiments show that our approach can achieve state-of-the-art on text recognition benchmarks.

cs.CV↗

arXiv2021

LUMINOUS: Indoor Scene Generation for Embodied AI Challenges

Learning-based methods for training embodied agents typically require a large number of high-quality scenes that contain realistic layouts and support meaningful interactions. However, current simulators for Embodied AI (EAI) challenges only provide simulated indoor scenes with a limited number of layouts. This paper presents Luminous, the first research framework that employs state-of-the-art indoor scene synthesis algorithms to generate large-scale simulated scenes for Embodied AI challenges. Further, we automatically and quantitatively evaluate the quality of generated indoor scenes via their ability to support complex household tasks. Luminous incorporates a novel scene generation algorithm (Constrained Stochastic Scene Generation (CSSG)), which achieves competitive performance with human-designed scenes. Within Luminous, the EAI task executor, task instruction generation module, and video rendering toolkit can collectively generate a massive multimodal dataset of new scenes for the training and evaluation of Embodied AI agents. Extensive experimental results demonstrate the effectiveness of the data generated by Luminous, enabling the comprehensive assessment of embodied agents on generalization and robustness.

cs.AI↗

arXiv2021

ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations

Object manipulation from 3D visual inputs poses many challenges on building generalizable perception and policy models. However, 3D assets in existing benchmarks mostly lack the diversity of 3D shapes that align with real-world intra-class complexity in topology and geometry. Here we propose SAPIEN Manipulation Skill Benchmark (ManiSkill) to benchmark manipulation skills over diverse objects in a full-physics simulator. 3D assets in ManiSkill include large intra-class topological and geometric variations. Tasks are carefully chosen to cover distinct types of manipulation challenges. Latest progress in 3D vision also makes us believe that we should customize the benchmark so that the challenge is inviting to researchers working on 3D deep learning. To this end, we simulate a moving panoramic camera that returns ego-centric point clouds or RGB-D images. In addition, we would like ManiSkill to serve a broad set of researchers interested in manipulation research. Besides supporting the learning of policies from interactions, we also support learning-from-demonstrations (LfD) methods, by providing a large number of high-quality demonstrations (~36,000 successful trajectories, ~1.5M point cloud/RGB-D frames in total). We provide baselines using 3D deep learning and LfD algorithms. All code of our benchmark (simulator, environment, SDK, and baselines) is open-sourced, and a challenge facing interdisciplinary researchers will be held based on the benchmark.

cs.LG↗

arXiv2021

Semantically Robust Unpaired Image Translation for Data with Unmatched Semantics Statistics

Many applications of unpaired image-to-image translation require the input contents to be preserved semantically during translations. Unaware of the inherently unmatched semantics distributions between source and target domains, existing distribution matching methods (i.e., GAN-based) can give undesired solutions. In particular, although producing visually reasonable outputs, the learned models usually flip the semantics of the inputs. To tackle this without using extra supervision, we propose to enforce the translated outputs to be semantically invariant w.r.t. small perceptual variations of the inputs, a property we call "semantic robustness". By optimizing a robustness loss w.r.t. multi-scale feature space perturbations of the inputs, our method effectively reduces semantics flipping and produces translations that outperform existing methods both quantitatively and qualitatively.

cs.CV↗

arXiv2021

0.75 Gbit/s high-speed classical key distribution with mode-shift keying chaos synchronization of Fabry-Perot lasers

High-speed physical key distribution is diligently pursued for secure communication. In this paper, we propose and experimentally demonstrate a scheme of high-speed key distribution using mode-shift keying chaos synchronization between two multi-longitudinal-mode Fabry-Perot lasers commonly driven by a super-luminescent diode. Legitimate users dynamically select one of the longitudinal modes according to private control codes to achieve mode-shift keying chaos synchronization. The two remote chaotic light waveforms are quantized to generate two raw random bit streams, and then those bits corresponding to chaos synchronization are sifted as shared keys by comparing the control codes. In this method, the transition time, i.e., the chaos synchronization recovery time is determined by the rising time of the control codes rather than the laser transition response time, so the key distribution rate is improved greatly. Our experiment achieved 0.75-Gbit/s key distribution rate with a bit error rate of 3.8*10-3 over 160-km fiber transmission with dispersion compensation. The entropy rate of the laser chaos is evaluated as 16 Gbit/s, which determines the ultimate final key rate together with key generation ratio. It is therefore believed that the method pays a way for Gbit/s physical key distribution.

physics.optics↗

arXiv2021

IFR: Iterative Fusion Based Recognizer For Low Quality Scene Text Recognition

Although recent works based on deep learning have made progress in improving recognition accuracy on scene text recognition, how to handle low-quality text images in end-to-end deep networks remains a research challenge. In this paper, we propose an Iterative Fusion based Recognizer (IFR) for low quality scene text recognition, taking advantage of refined text images input and robust feature representation. IFR contains two branches which focus on scene text recognition and low quality scene text image recovery respectively. We utilize an iterative collaboration between two branches, which can effectively alleviate the impact of low quality input. A feature fusion module is proposed to strengthen the feature representation of the two branches, where the features from the Recognizer are Fused with image Restoration branch, referred to as RRF. Without changing the recognition network structure, extensive quantitative and qualitative experimental results show that the proposed method significantly outperforms the baseline methods in boosting the recognition accuracy of benchmark datasets and low resolution images in TextZoom dataset.

cs.CV↗

arXiv2021

Tracking Based Semi-Automatic Annotation for Scene Text Videos

Recently, video scene text detection has received increasing attention due to its comprehensive applications. However, the lack of annotated scene text video datasets has become one of the most important problems, which hinders the development of video scene text detection. The existing scene text video datasets are not large-scale due to the expensive cost caused by manual labeling. In addition, the text instances in these datasets are too clear to be a challenge. To address the above issues, we propose a tracking based semi-automatic labeling strategy for scene text videos in this paper. We get semi-automatic scene text annotation by labeling manually for the first frame and tracking automatically for the subsequent frames, which avoid the huge cost of manual labeling. Moreover, a paired low-quality scene text video dataset named Text-RBL is proposed, consisting of raw videos, blurry videos, and low-resolution videos, labeled by the proposed convenient semi-automatic labeling strategy. Through an averaging operation and bicubic down-sampling operation over the raw videos, we can efficiently obtain blurry videos and low-resolution videos paired with raw videos separately. To verify the effectiveness of Text-RBL, we propose a baseline model combined with the text detector and tracker for video scene text detection. Moreover, a failure detection scheme is designed to alleviate the baseline model drift issue caused by complex scenes. Extensive experiments demonstrate that Text-RBL with paired low-quality videos labeled by the semi-automatic method can significantly improve the performance of the text detector in low-quality scenes.

cs.CV↗

arXiv2020

Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals

We study how to learn a policy with compositional generalizability. We propose a two-stage framework, which refactorizes a high-reward teacher policy into a generalizable student policy with strong inductive bias. Particularly, we implement an object-centric GNN-based student policy, whose input objects are learned from images through self-supervised learning. Empirically, we evaluate our approach on four difficult tasks that require compositional generalizability, and achieve superior performance compared to baselines.

cs.CV↗