Searcharxiv⌕ Search

arXiv subjects

Dong Wang

Publications and source records attributed to Dong Wang.

At least 451 records · Page 25Linked to original sources
arXiv2023

One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field

Talking head generation aims to generate faces that maintain the identity information of the source image and imitate the motion of the driving image. Most pioneering methods rely primarily on 2D representations and thus will inevitably suffer from face distortion when large head rotations are encountered. Recent works instead employ explicit 3D structural representations or implicit neural rendering to improve performance under large pose changes. Nevertheless, the fidelity of identity and expression is not so desirable, especially for novel-view synthesis. In this paper, we propose HiDe-NeRF, which achieves high-fidelity and free-view talking-head synthesis. Drawing on the recently proposed Deformable Neural Radiance Fields, HiDe-NeRF represents the 3D dynamic scene into a canonical appearance field and an implicit deformation field, where the former comprises the canonical source face and the latter models the driving pose and expression. In particular, we improve fidelity from two aspects: (i) to enhance identity expressiveness, we design a generalized appearance module that leverages multi-scale volume features to preserve face shape and details; (ii) to improve expression preciseness, we propose a lightweight deformation module that explicitly decouples the pose and expression to enable precise expression modeling. Extensive experiments demonstrate that our proposed approach can generate better results than previous works. Project page: https://www.waytron.net/hidenerf/

cs.CV↗
arXiv2023

Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement

The popularity of Contrastive Language-Image Pre-training (CLIP) has propelled its application to diverse downstream vision tasks. To improve its capacity on downstream tasks, few-shot learning has become a widely-adopted technique. However, existing methods either exhibit limited performance or suffer from excessive learnable parameters. In this paper, we propose APE, an Adaptive Prior rEfinement method for CLIP's pre-trained knowledge, which achieves superior accuracy with high computational efficiency. Via a prior refinement module, we analyze the inter-class disparity in the downstream data and decouple the domain-specific knowledge from the CLIP-extracted cache model. On top of that, we introduce two model variants, a training-free APE and a training-required APE-T. We explore the trilateral affinities between the test image, prior cache model, and textual representations, and only enable a lightweight category-residual module to be trained. For the average accuracy over 11 benchmarks, both APE and APE-T attain state-of-the-art and respectively outperform the second-best by +1.59% and +1.99% under 16 shots with x30 less learnable parameters.

cs.CV↗
arXiv2023

Understanding the Role of Images on Stack Overflow

Images are increasingly being shared by software developers in diverse channels including question-and-answer forums like Stack Overflow. Although prior work has pointed out that these images are meaningful and provide complementary information compared to their associated text, how images are used to support questions is empirically unknown. To address this knowledge gap, in this paper we specifically conduct an empirical study to investigate (I) the characteristics of images, (II) the extent to which images are used in different question types, and (III) the role of images on receiving answers. Our results first show that user interface is the most common image content and undesired output is the most frequent purpose for sharing images. Moreover, these images essentially facilitate the understanding of 68% of sampled questions. Second, we find that discrepancy questions are more relatively frequent compared to those without images, but there are no significant differences observed in description length in all types of questions. Third, the quantitative results statistically validate that questions with images are more likely to receive accepted answers, but do not speed up the time to receive answers. Our work demonstrates the crucial role that images play by approaching the topic from a new angle and lays the foundation for future opportunities to use images to assist in tasks like generating questions and identifying question-relatedness.

cs.SE↗
arXiv2023

Fully Self-Supervised Depth Estimation from Defocus Clue

Depth-from-defocus (DFD), modeling the relationship between depth and defocus pattern in images, has demonstrated promising performance in depth estimation. Recently, several self-supervised works try to overcome the difficulties in acquiring accurate depth ground-truth. However, they depend on the all-in-focus (AIF) images, which cannot be captured in real-world scenarios. Such limitation discourages the applications of DFD methods. To tackle this issue, we propose a completely self-supervised framework that estimates depth purely from a sparse focal stack. We show that our framework circumvents the needs for the depth and AIF image ground-truth, and receives superior predictions, thus closing the gap between the theoretical success of DFD works and their applications in the real world. In particular, we propose (i) a more realistic setting for DFD tasks, where no depth or AIF image ground-truth is available; (ii) a novel self-supervision framework that provides reliable predictions of depth and AIF image under the challenging setting. The proposed framework uses a neural model to predict the depth and AIF image, and utilizes an optical model to validate and refine the prediction. We verify our framework on three benchmark datasets with rendered focal stacks and real focal stacks. Qualitative and quantitative evaluations show that our method provides a strong baseline for self-supervised DFD tasks.

eess.IV↗
arXiv2023

Propagate And Calibrate: Real-time Passive Non-line-of-sight Tracking

Non-line-of-sight (NLOS) tracking has drawn increasing attention in recent years, due to its ability to detect object motion out of sight. Most previous works on NLOS tracking rely on active illumination, e.g., laser, and suffer from high cost and elaborate experimental conditions. Besides, these techniques are still far from practical application due to oversimplified settings. In contrast, we propose a purely passive method to track a person walking in an invisible room by only observing a relay wall, which is more in line with real application scenarios, e.g., security. To excavate imperceptible changes in videos of the relay wall, we introduce difference frames as an essential carrier of temporal-local motion messages. In addition, we propose PAC-Net, which consists of alternating propagation and calibration, making it capable of leveraging both dynamic and static messages on a frame-level granularity. To evaluate the proposed method, we build and publish the first dynamic passive NLOS tracking dataset, NLOS-Track, which fills the vacuum of realistic NLOS datasets. NLOS-Track contains thousands of NLOS video clips and corresponding trajectories. Both real-shot and synthetic data are included. Our codes and dataset are available at https://againstentropy.github.io/NLOS-Track/.

cs.CV↗
arXiv2023

Visual Prompt Multi-Modal Tracking

Visible-modal object tracking gives rise to a series of downstream multi-modal tracking tributaries. To inherit the powerful representations of the foundation model, a natural modus operandi for multi-modal tracking is full fine-tuning on the RGB-based parameters. Albeit effective, this manner is not optimal due to the scarcity of downstream data and poor transferability, etc. In this paper, inspired by the recent success of the prompt learning in language models, we develop Visual Prompt multi-modal Tracking (ViPT), which learns the modal-relevant prompts to adapt the frozen pre-trained foundation model to various downstream multimodal tracking tasks. ViPT finds a better way to stimulate the knowledge of the RGB-based model that is pre-trained at scale, meanwhile only introducing a few trainable parameters (less than 1% of model parameters). ViPT outperforms the full fine-tuning paradigm on multiple downstream tracking tasks including RGB+Depth, RGB+Thermal, and RGB+Event tracking. Extensive experiments show the potential of visual prompt learning for multi-modal tracking, and ViPT can achieve state-of-the-art performance while satisfying parameter efficiency. Code and models are available at https://github.com/jiawen-zhu/ViPT.

cs.CV↗
arXiv2023

Tessellated granular metamaterials with tunable elastic moduli

Most granular packings possess shear moduli ($G$) that increase with the applied external pressure, and bulk moduli ($B$) that increase or remain constant with pressure. This paper presents "tessellated" granular metamaterials for which both $G$ and the ratio $G/B$ decrease with increasing pressure. The granular metamaterials are made from flexible tessellations forming a ring of closed cells, each containing a small number of solid particles. For under-constrained tessellations, the dominant contributions to $G$ and $B$ are the particle-particle and particle-cell interactions. With specific particle configurations in the cells, we limit the number of possible particle rearrangements to achieve decreasing $G$ as we increase the pressure difference between the inside and outside of the tessellation, leading to $G/B \ll 1$ at large pressures. We further study tessellated granular metamaterials with cells containing a single particle and many particles to determine the variables that control the mechanical response of particle-filled tessellations as a function of pressure.

cond-mat.soft↗
arXiv2023

Dual Memory Aggregation Network for Event-Based Object Detection with Learnable Representation

Event-based cameras are bio-inspired sensors that capture brightness change of every pixel in an asynchronous manner. Compared with frame-based sensors, event cameras have microsecond-level latency and high dynamic range, hence showing great potential for object detection under high-speed motion and poor illumination conditions. Due to sparsity and asynchronism nature with event streams, most of existing approaches resort to hand-crafted methods to convert event data into 2D grid representation. However, they are sub-optimal in aggregating information from event stream for object detection. In this work, we propose to learn an event representation optimized for event-based object detection. Specifically, event streams are divided into grids in the x-y-t coordinates for both positive and negative polarity, producing a set of pillars as 3D tensor representation. To fully exploit information with event streams to detect objects, a dual-memory aggregation network (DMANet) is proposed to leverage both long and short memory along event streams to aggregate effective information for object detection. Long memory is encoded in the hidden state of adaptive convLSTMs while short memory is modeled by computing spatial-temporal correlation between event pillars at neighboring time intervals. Extensive experiments on the recently released event-based automotive detection dataset demonstrate the effectiveness of the proposed method.

cs.CV↗
arXiv2023

A Survey on Social-Physical Sensing: An Emerging Sensing Paradigm that Explores the Collective Intelligence of Humans and Machine

Propelled by the omnipresence of versatile data capture, communication, and computing technologies, physical sensing has revolutionized the avenue for decisively interpreting the real world. However, various limitations hinder physical sensing's effectiveness in critical scenarios such as disaster response and urban anomaly detection. Meanwhile, social sensing is contriving as a pervasive sensing paradigm leveraging observations from human participants equipped with portable devices and ubiquitous Internet connectivity to perceive the environment. Despite its virtues, social sensing also inherently suffers from a few drawbacks (e.g., inconsistent reliability and uncertain data provenance). Motivated by the complementary strengths of the two sensing modes, social-physical sensing (SPS) is protruding as an emerging sensing paradigm that explores the collective intelligence of humans and machines to reconstruct the "state of the world", both physically and socially. While a good number of interesting SPS applications have been studied, several critical unsolved challenges still exist in SPS. In this paper, we provide a comprehensive survey of SPS, emphasizing its definition, key enablers, state-of-the-art applications, potential research challenges, and roadmap for future work. This paper intends to bridge the knowledge gap of existing sensing-focused survey papers by thoroughly examining the various aspects of SPS crucial for building potent SPS systems.

cs.SI↗
arXiv2023

Holographic Entanglement Entropy and Complexity for D-Wave Superconductors

By using the RT formula and the subregion CV conjecture respectively, we numerically investigate the holographic entanglement entropy (HEE) and holographic subregion complexity (HSC) for two holographic d-wave superconducting models with backreactions. We find that both the HEE and HSC can be used as good probes to these two d-wave superconducting phase transitions. The HEE of superconducting phase is always lower than that of the normal phase due to the condensation of degrees of freedom below the critical temperature Tc. However, for the HSC, it behaves differently and interestingly, which depends on both the strip-width Lx and backreaction \k{appa}.

hep-th↗
arXiv2023

Ballistic Ejection of Microdroplets from Overpacked Interfacial Assemblies

Spontaneous emulsification, resulting from the assembly and accumulation of surfactants at liquid-liquid interfaces, is an interfacial instability where microdroplets are generated and diffusively spread from the interface until complete emulsification. Here, we show that an external magnetic field can modulate the assembly of paramagnetic nanoparticle surfactants (NPSs) at liquid-liquid interfaces and trigger an oversaturation in the areal density of the NPSs at the interface, as evidenced by a marked reduction in the interfacial tension, γ, and corroborated with a magnetostatic continuum theory. Despite the significant reduction in γ, the presence of the magnetic field does not cause stable interfaces to become unstable. Upon rapid removal of the field, however, an explosive ejection of a plume of microdroplets from the surface occurs, a dynamical interfacial instability which is termed explosive emulsification. This explosive event rapidly reduces the areal density of the NPSs to its pre-field level, stabilizing the interface. The ability to externally suppress or trigger the explosive emulsification and controlled generation of tens of thousands of microdroplets, uncovers an efficient energy storage and release process, that has potential applications for controlled and directed delivery of chemicals and remotely controlled soft microrobots, taking advantage of the ferromagnetic nature of the microdroplets.

cond-mat.soft↗
arXiv2023

Multiscale Latent-Guided Entropy Model for LiDAR Point Cloud Compression

The non-uniform distribution and extremely sparse nature of the LiDAR point cloud (LPC) bring significant challenges to its high-efficient compression. This paper proposes a novel end-to-end, fully-factorized deep framework that encodes the original LPC into an octree structure and hierarchically decomposes the octree entropy model in layers. The proposed framework utilizes a hierarchical latent variable as side information to encapsulate the sibling and ancestor dependence, which provides sufficient context information for the modelling of point cloud distribution while enabling the parallel encoding and decoding of octree nodes in the same layer. Besides, we propose a residual coding framework for the compression of the latent variable, which explores the spatial correlation of each layer by progressive downsampling, and model the corresponding residual with a fully-factorized entropy model. Furthermore, we propose soft addition and subtraction for residual coding to improve network flexibility. The comprehensive experiment results on the LiDAR benchmark SemanticKITTI and MPEG-specified dataset Ford demonstrates that our proposed framework achieves state-of-the-art performance among all the previous LPC frameworks. Besides, our end-to-end, fully-factorized framework is proved by experiment to be high-parallelized and time-efficient and saves more than 99.8% of decoding time compared to previous state-of-the-art methods on LPC compression.

cs.CV↗
arXiv2023

PIER: Permutation-Level Interest-Based End-to-End Re-ranking Framework in E-commerce

Re-ranking draws increased attention on both academics and industries, which rearranges the ranking list by modeling the mutual influence among items to better meet users' demands. Many existing re-ranking methods directly take the initial ranking list as input, and generate the optimal permutation through a well-designed context-wise model, which brings the evaluation-before-reranking problem. Meanwhile, evaluating all candidate permutations brings unacceptable computational costs in practice. Thus, to better balance efficiency and effectiveness, online systems usually use a two-stage architecture which uses some heuristic methods such as beam-search to generate a suitable amount of candidate permutations firstly, which are then fed into the evaluation model to get the optimal permutation. However, existing methods in both stages can be improved through the following aspects. As for generation stage, heuristic methods only use point-wise prediction scores and lack an effective judgment. As for evaluation stage, most existing context-wise evaluation models only consider the item context and lack more fine-grained feature context modeling. This paper presents a novel end-to-end re-ranking framework named PIER to tackle the above challenges which still follows the two-stage architecture and contains two mainly modules named FPSM and OCPM. We apply SimHash in FPSM to select top-K candidates from the full permutation based on user's permutation-level interest in an efficient way. Then we design a novel omnidirectional attention mechanism in OCPM to capture the context information in the permutation. Finally, we jointly train these two modules end-to-end by introducing a comparative learning loss. Offline experiment results demonstrate that PIER outperforms baseline models on both public and industrial datasets, and we have successfully deployed PIER on Meituan food delivery platform.

cs.IR↗
arXiv2023

A Deep Behavior Path Matching Network for Click-Through Rate Prediction

User behaviors on an e-commerce app not only contain different kinds of feedback on items but also sometimes imply the cognitive clue of the user's decision-making. For understanding the psychological procedure behind user decisions, we present the behavior path and propose to match the user's current behavior path with historical behavior paths to predict user behaviors on the app. Further, we design a deep neural network for behavior path matching and solve three difficulties in modeling behavior paths: sparsity, noise interference, and accurate matching of behavior paths. In particular, we leverage contrastive learning to augment user behavior paths, provide behavior path self-activation to alleviate the effect of noise, and adopt a two-level matching mechanism to identify the most appropriate candidate. Our model shows excellent performance on two real-world datasets, outperforming the state-of-the-art CTR model. Moreover, our model has been deployed on the Meituan food delivery platform and has accumulated 1.6% improvement in CTR and 1.8% improvement in advertising revenue.

cs.AI↗
arXiv2023

Decision-Making Context Interaction Network for Click-Through Rate Prediction

Click-through rate (CTR) prediction is crucial in recommendation and online advertising systems. Existing methods usually model user behaviors, while ignoring the informative context which influences the user to make a click decision, e.g., click pages and pre-ranking candidates that inform inferences about user interests, leading to suboptimal performance. In this paper, we propose a Decision-Making Context Interaction Network (DCIN), which deploys a carefully designed Context Interaction Unit (CIU) to learn decision-making contexts and thus benefits CTR prediction. In addition, the relationship between different decision-making context sources is explored by the proposed Adaptive Interest Aggregation Unit (AIAU) to improve CTR prediction further. In the experiments on public and industrial datasets, DCIN significantly outperforms the state-of-the-art methods. Notably, the model has obtained the improvement of CTR+2.9%/CPM+2.1%/GMV+1.5% for online A/B testing and served the main traffic of Meituan Waimai advertising system.

cs.IR↗
arXiv2023

Efficient and robust chiral discrimination by invariant-based inverse engineering

We propose an accurate and convenient method to achieve 100\% discrimination of chiral molecules with Lewis-Riesenfeld invariant. By reversely designing the pulse scheme of handed resolution, we obtain the parameters of the three-level Hamiltonians to achieve this goal. For the same initial state, we can completely transfer its population to one energy level for left-handed molecules, while transfer it to another energy level for right-handed molecules. Moreover, this method can be further optimized when errors exist, and it shows that the optimal method are more robust against these errors than the counterdiabatic and original invariant-based shortcut schemes. This provides an effective, accurate, and robust method to distinguish the handedness of molecules.

quant-ph↗
arXiv2022

Trade-off relations of quantum resource theory in neutrino oscillations

The violation of the classical bounds imposed by Leggett-Garg inequalities has tested the quantumness of neutrino oscillations (NOs) over a long distance during the propagation. The measure of quantumness in experimentally observed NOs is studied via quantum resource theory (QRT). Here, we focus on the trade-off relations of QRT in the three-flavor NOs, based on Bell-type violations, first-order coherence and intrinsic concurrence, and the relative entropy of coherence. For the electron and muon antineutrino oscillations, the analytical trade-off relations obeyed by the Bell-CHSH inequality of pairwise flavor states in this three-flavor neutrino system are obtained; the sum of the maximal violation of the CHSH tests for three pairwise flavor states is less than or equal to 12. Moreover, there exists an equality relation concerning first-order coherence and intrinsic concurrence in NOs, showing how much quantum resources flow between first-order coherence and intrinsic concurrence during the neutrino propagation. In addition, it is found that the tripartite coherence of three-flavor system is equal to or larger than the sum of the coherence of reduced bipartite flavor states. The trade-off relations of QRT provide a method for studying how the quantum resources convert and distribute in NOs, which might inspire the future applications in quantum information processing using neutrinos.

quant-ph↗
arXiv2022

Lyapunov exponent, universality and phase transition for products of random matrices

Products of $M$ i.i.d. random matrices of size $N \times N$ are related to classical limit theorems in probability theory ($N=1$ and large $M$), to Lyapunov exponents in dynamical systems (finite $N$ and large $M$), and to universality in random matrix theory (finite $M$ and large $N$). Under the two different limits of $M \to \infty$ and $N \to \infty$, the local singular value statistics display Gaussian and random matrix theory universality, respectively. However, it is unclear what happens if both $M$ and $N$ go to infinity. This problem, proposed by Akemann, Burda, Kieburg \cite{Akemann-Burda-Kieburg14} and Deift \cite{Deift17}, lies at the heart of understanding both kinds of universal limits. In the case of complex Gaussian random matrices, we prove that there exists a crossover phenomenon as the relative ratio of $M$ and $N$ changes from $0$ to $\infty$: sine and Airy kernels from the Gaussian Unitary Ensemble (GUE) when $M/N \to 0$, Gaussian fluctuation when $M/N \to \infty$, and new critical phenomena when $M/N \to γ\in (0,\infty)$. Accordingly, we further prove that the largest singular value undergoes a phase transition between the Gaussian and GUE Tracy-Widom distributions.

math.PR↗