arXiv subjects

Fu Li

Publications and source records attributed to Fu Li.

At least 73 records · Page 4Linked to original sources

arXiv2022

Boosting Video-Text Retrieval with Explicit High-Level Semantics

Video-text retrieval (VTR) is an attractive yet challenging task for multi-modal understanding, which aims to search for relevant video (text) given a query (video). Existing methods typically employ completely heterogeneous visual-textual information to align video and text, whilst lacking the awareness of homogeneous high-level semantic information residing in both modalities. To fill this gap, in this work, we propose a novel visual-linguistic aligning model named HiSE for VTR, which improves the cross-modal representation by incorporating explicit high-level semantics. First, we explore the hierarchical property of explicit high-level semantics, and further decompose it into two levels, i.e. discrete semantics and holistic semantics. Specifically, for visual branch, we exploit an off-the-shelf semantic entity predictor to generate discrete high-level semantics. In parallel, a trained video captioning model is employed to output holistic high-level semantics. As for the textual modality, we parse the text into three parts including occurrence, action and entity. In particular, the occurrence corresponds to the holistic high-level semantics, meanwhile both action and entity represent the discrete ones. Then, different graph reasoning techniques are utilized to promote the interaction between holistic and discrete high-level semantics. Extensive experiments demonstrate that, with the aid of explicit high-level semantics, our method achieves the superior performance over state-of-the-art methods on three benchmark datasets, including MSR-VTT, MSVD and DiDeMo.

cs.CV↗

arXiv2022

Egalitarian Resource Sharing Over Multiple Rounds

It is often beneficial for agents to pool their resources in order to better accommodate fluctuations in individual demand. Many multi-round resource allocation mechanisms operate in an online manner: in each round, the agents specify their demands for that round, and the mechanism determines a corresponding allocation. In this paper, we focus instead on the offline setting in which the agents specify their demand for each round at the outset. We formulate a specific resource allocation problem in this setting, and design and analyze an associated mechanism based on the solution concept of lexicographic maximin fairness. We present an efficient implementation of our mechanism, and prove that it is envy-free, non-wasteful, resource monotonic, population monotonic, and group strategyproof. We also prove that our mechanism guarantees each agent at least half of the utility that they can obtain by not sharing their resources. We complement these positive results by proving that no maximin fair mechanism can improve on the aforementioned factor of one-half.

cs.GT↗

arXiv2022

Quantum-Enhanced Stimulated Brillouin Scattering Spectroscopy and Imaging

Brillouin microscopy is an emerging label-free imaging technique to assess local viscoelastic properties. Quantum-enhanced stimulated Brillouin scattering is demonstrated for the first time using low power continuous-wave lasers at 795~nm. A signal to noise ratio enhancement of 3.4~dB is reported by using two-mode intensity-difference squeezed light generated with the four-wave mixing process in atomic rubidium vapor. The low optical power and the excitation wavelengths in the water transparency window has the potential to provide a powerful bio-imaging technique for probing mechanical properties of biological samples prone to phototoxicity and thermal effects. The performance enhancement affordable through the use of quantum light may pave the way for significantly improved sensitivity that cannot be achieved classically. The proposed new way of utilizing squeezed light for enhanced stimulated Brillouin scattering can be easily adapted for both spectroscopic and imaging applications in materials science and biology.

physics.optics↗

arXiv2022

3-D Stochastic Numerical Breast Phantoms for Enabling Virtual Imaging Trials of Ultrasound Computed Tomography

Ultrasound computed tomography (USCT) is an emerging imaging modality for breast imaging that can produce quantitative images that depict the acoustic properties of tissues. Computer-simulation studies, also known as virtual imaging trials, provide researchers with an economical and convenient route to systematically explore imaging system designs and image reconstruction methods. When simulating an imaging technology intended for clinical use, it is essential to employ realistic numerical phantoms that can facilitate the objective, or task-based, assessment of image quality. Moreover, when computing objective image quality measures, an ensemble of such phantoms should be employed that display the variability in anatomy and object properties that is representative of the to-be-imaged patient cohort. Such stochastic phantoms for clinically relevant applications of USCT are currently lacking. In this work, a methodology for producing realistic three-dimensional (3D) numerical breast phantoms for enabling clinically relevant computer-simulation studies of USCT breast imaging is presented. By extending and adapting an existing stochastic 3D breast phantom for use withUSCT, methods for creating ensembles of numerical acoustic breast phantoms are established. These breast phantoms will possess clinically relevant variations in breast size, composition, acoustic properties, tumor locations, and tissue textures. To demonstrate the use of the phantoms in virtual USCT studies, two brief case studies are presented that address the development and assessment of image reconstruction procedures. Examples of breast phantoms produced by use of the proposed methods and a collection of 52 sets of simulated USCT measurement data have been made open source for use in image reconstruction development

physics.med-ph↗

arXiv2022

GMSS: Graph-Based Multi-Task Self-Supervised Learning for EEG Emotion Recognition

Previous electroencephalogram (EEG) emotion recognition relies on single-task learning, which may lead to overfitting and learned emotion features lacking generalization. In this paper, a graph-based multi-task self-supervised learning model (GMSS) for EEG emotion recognition is proposed. GMSS has the ability to learn more general representations by integrating multiple self-supervised tasks, including spatial and frequency jigsaw puzzle tasks, and contrastive learning tasks. By learning from multiple tasks simultaneously, GMSS can find a representation that captures all of the tasks thereby decreasing the chance of overfitting on the original task, i.e., emotion recognition task. In particular, the spatial jigsaw puzzle task aims to capture the intrinsic spatial relationships of different brain regions. Considering the importance of frequency information in EEG emotional signals, the goal of the frequency jigsaw puzzle task is to explore the crucial frequency bands for EEG emotion recognition. To further regularize the learned features and encourage the network to learn inherent representations, contrastive learning task is adopted in this work by mapping the transformed data into a common feature space. The performance of the proposed GMSS is compared with several popular unsupervised and supervised methods. Experiments on SEED, SEED-IV, and MPED datasets show that the proposed model has remarkable advantages in learning more discriminative and general features for EEG emotional signals.

eess.SP↗

arXiv2022

OSOP: A Multi-Stage One Shot Object Pose Estimation Framework

We present a novel one-shot method for object detection and 6 DoF pose estimation, that does not require training on target objects. At test time, it takes as input a target image and a textured 3D query model. The core idea is to represent a 3D model with a number of 2D templates rendered from different viewpoints. This enables CNN-based direct dense feature extraction and matching. The object is first localized in 2D, then its approximate viewpoint is estimated, followed by dense 2D-3D correspondence prediction. The final pose is computed with PnP. We evaluate the method on LineMOD, Occlusion, Homebrewed, YCB-V and TLESS datasets and report very competitive performance in comparison to the state-of-the-art methods trained on synthetic data, even though our method is not trained on the object models used for testing.

cs.CV↗

arXiv2022

Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model

To achieve disentangled image manipulation, previous works depend heavily on manual annotation. Meanwhile, the available manipulations are limited to a pre-defined set the models were trained for. We propose a novel framework, i.e., Predict, Prevent, and Evaluate (PPE), for disentangled text-driven image manipulation that requires little manual annotation while being applicable to a wide variety of manipulations. Our method approaches the targets by deeply exploiting the power of the large-scale pre-trained vision-language model CLIP. Concretely, we firstly Predict the possibly entangled attributes for a given text command. Then, based on the predicted attributes, we introduce an entanglement loss to Prevent entanglements during training. Finally, we propose a new evaluation metric to Evaluate the disentangled image manipulation. We verify the effectiveness of our method on the challenging face editing task. Extensive experiments show that the proposed PPE framework achieves much better quantitative and qualitative results than the up-to-date StyleCLIP baseline.

cs.CV↗

arXiv2021

Progressive Graph Convolution Network for EEG Emotion Recognition

Studies in the area of neuroscience have revealed the relationship between emotional patterns and brain functional regions, demonstrating that dynamic relationships between different brain regions are an essential factor affecting emotion recognition determined through electroencephalography (EEG). Moreover, in EEG emotion recognition, we can observe that clearer boundaries exist between coarse-grained emotions than those between fine-grained emotions, based on the same EEG data; this indicates the concurrence of large coarse- and small fine-grained emotion variations. Thus, the progressive classification process from coarse- to fine-grained categories may be helpful for EEG emotion recognition. Consequently, in this study, we propose a progressive graph convolution network (PGCN) for capturing this inherent characteristic in EEG emotional signals and progressively learning the discriminative EEG features. To fit different EEG patterns, we constructed a dual-graph module to characterize the intrinsic relationship between different EEG channels, containing the dynamic functional connections and static spatial proximity information of brain regions from neuroscience research. Moreover, motivated by the observation of the relationship between coarse- and fine-grained emotions, we adopt a dual-head module that enables the PGCN to progressively learn more discriminative EEG features, from coarse-grained (easy) to fine-grained categories (difficult), referring to the hierarchical characteristic of emotion. To verify the performance of our model, extensive experiments were conducted on two public datasets: SEED-IV and multi-modal physiological emotion database (MPED).

cs.CV↗

arXiv2021

CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration

We study the problem of extracting correspondences between a pair of point clouds for registration. For correspondence retrieval, existing works benefit from matching sparse keypoints detected from dense points but usually struggle to guarantee their repeatability. To address this issue, we present CoFiNet - Coarse-to-Fine Network which extracts hierarchical correspondences from coarse to fine without keypoint detection. On a coarse scale and guided by a weighting scheme, our model firstly learns to match down-sampled nodes whose vicinity points share more overlap, which significantly shrinks the search space of a consecutive stage. On a finer scale, node proposals are consecutively expanded to patches that consist of groups of points together with associated descriptors. Point correspondences are then refined from the overlap areas of corresponding patches, by a density-adaptive matching module capable to deal with varying point density. Extensive evaluation of CoFiNet on both indoor and outdoor standard benchmarks shows our superiority over existing methods. Especially on 3DLoMatch where point clouds share less overlap, CoFiNet significantly outperforms state-of-the-art approaches by at least 5% on Registration Recall, with at most two-third of their parameters.

cs.CV↗

arXiv2021

Object Allocation Over a Network of Objects: Mobile Agents with Strict Preferences

In recent work, Gourvès, Lesca, and Wilczynski propose a variant of the classic housing markets model where the matching between agents and objects evolves through Pareto-improving swaps between pairs of adjacent agents in a social network. To explore the swap dynamics of their model, they pose several basic questions concerning the set of reachable matchings. In their work and other follow-up works, these questions have been studied for various classes of graphs: stars, paths, generalized stars (i.e., trees where at most one vertex has degree greater than two), trees, and cliques. For generalized stars and trees, it remains open whether a Pareto-efficient reachable matching can be found in polynomial time. In this paper, we pursue the same set of questions under a natural variant of their model. In our model, the social network is replaced by a network of objects, and a swap is allowed to take place between two agents if it is Pareto-improving and the associated objects are adjacent in the network. In those cases where the question of polynomial-time solvability versus NP-hardness has been resolved for the social network model, we are able to show that the same result holds for the network-of-objects model. In addition, for our model, we present a polynomial-time algorithm for computing a Pareto-efficient reachable matching in generalized star networks. Moreover, the object reachability algorithm that we present for path networks is significantly faster than the known polynomial-time algorithms for the same question in the social network model.

cs.GT↗

arXiv2021

DOLG: Single-Stage Image Retrieval with Deep Orthogonal Fusion of Local and Global Features

Image Retrieval is a fundamental task of obtaining images similar to the query one from a database. A common image retrieval practice is to firstly retrieve candidate images via similarity search using global image features and then re-rank the candidates by leveraging their local features. Previous learning-based studies mainly focus on either global or local image representation learning to tackle the retrieval task. In this paper, we abandon the two-stage paradigm and seek to design an effective single-stage solution by integrating local and global information inside images into compact image representations. Specifically, we propose a Deep Orthogonal Local and Global (DOLG) information fusion framework for end-to-end image retrieval. It attentively extracts representative local information with multi-atrous convolutions and self-attention at first. Components orthogonal to the global image representation are then extracted from the local information. At last, the orthogonal components are concatenated with the global representation as a complementary, and then aggregation is performed to generate the final representation. The whole framework is end-to-end differentiable and can be trained with image-level labels. Extensive experimental results validate the effectiveness of our solution and show that our model achieves state-of-the-art image retrieval performances on Revisited Oxford and Paris datasets.

cs.CV↗

arXiv2021

AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer

Fast arbitrary neural style transfer has attracted widespread attention from academic, industrial and art communities due to its flexibility in enabling various applications. Existing solutions either attentively fuse deep style feature into deep content feature without considering feature distributions, or adaptively normalize deep content feature according to the style such that their global statistics are matched. Although effective, leaving shallow feature unexplored and without locally considering feature statistics, they are prone to unnatural output with unpleasing local distortions. To alleviate this problem, in this paper, we propose a novel attention and normalization module, named Adaptive Attention Normalization (AdaAttN), to adaptively perform attentive normalization on per-point basis. Specifically, spatial attention score is learnt from both shallow and deep features of content and style images. Then per-point weighted statistics are calculated by regarding a style feature point as a distribution of attention-weighted output of all style feature points. Finally, the content feature is normalized so that they demonstrate the same local feature statistics as the calculated per-point weighted style feature statistics. Besides, a novel local feature loss is derived based on AdaAttN to enhance local visual quality. We also extend AdaAttN to be ready for video style transfer with slight modifications. Experiments demonstrate that our method achieves state-of-the-art arbitrary image/video style transfer. Codes and models are available.

cs.CV↗

arXiv2021

Paint Transformer: Feed Forward Neural Painting with Stroke Prediction

Neural painting refers to the procedure of producing a series of strokes for a given image and non-photo-realistically recreating it using neural networks. While reinforcement learning (RL) based agents can generate a stroke sequence step by step for this task, it is not easy to train a stable RL agent. On the other hand, stroke optimization methods search for a set of stroke parameters iteratively in a large search space; such low efficiency significantly limits their prevalence and practicality. Different from previous methods, in this paper, we formulate the task as a set prediction problem and propose a novel Transformer-based framework, dubbed Paint Transformer, to predict the parameters of a stroke set with a feed forward network. This way, our model can generate a set of strokes in parallel and obtain the final painting of size 512 * 512 in near real time. More importantly, since there is no dataset available for training the Paint Transformer, we devise a self-training pipeline such that it can be trained without any off-the-shelf dataset while still achieving excellent generalization capability. Experiments demonstrate that our method achieves better painting performance than previous ones with cheaper training and inference costs. Codes and models are available.

cs.CV↗

arXiv2021

Experimental study of decoherence of the two-mode squeezed vacuum state via second harmonic generation

Decoherence remains one of the most serious challenges to the implementation of quantum technology. It appears as a result of the transformation over time of a quantum superposition state into a classical mixture due to the quantum system interacting with the environment. Since quantum systems are never completely isolated from their environment, decoherence therefore cannot be avoided in realistic situations. Decoherence has been extensively studied, mostly theoretically, because it has many important implications in quantum technology, such as in the fields of quantum information processing, quantum communication and quantum computation. Here we report a novel experimental scheme on the study of decoherence of a two-mode squeezed vacuum state via its second harmonic generation signal. Our scheme can directly extract the decoherence of the phase-sensitive quantum correlation $\langle \hat{a}\hat{b}\rangle$ between two entangled modes $a$ and $b$. Such a correlation is the most important characteristic of a two-mode squeezed state. More importantly, this is an experimental study on the decoherence effect of a squeezed vacuum state, which has been rarely investigated.

physics.optics↗

arXiv2021

Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness

Inpainting arbitrary missing regions is challenging because learning valid features for various masked regions is nontrivial. Though U-shaped encoder-decoder frameworks have been witnessed to be successful, most of them share a common drawback of mask unawareness in feature extraction because all convolution windows (or regions), including those with various shapes of missing pixels, are treated equally and filtered with fixed learned kernels. To this end, we propose our novel mask-aware inpainting solution. Firstly, a Mask-Aware Dynamic Filtering (MADF) module is designed to effectively learn multi-scale features for missing regions in the encoding phase. Specifically, filters for each convolution window are generated from features of the corresponding region of the mask. The second fold of mask awareness is achieved by adopting Point-wise Normalization (PN) in our decoding phase, considering that statistical natures of features at masked points differentiate from those of unmasked points. The proposed PN can tackle this issue by dynamically assigning point-wise scaling factor and bias. Lastly, our model is designed to be an end-to-end cascaded refinement one. Supervision information such as reconstruction loss, perceptual loss and total variation loss is incrementally leveraged to boost the inpainting results from coarse to fine. Effectiveness of the proposed framework is validated both quantitatively and qualitatively via extensive experiments on three public datasets including Places2, CelebA and Paris StreetView.

cs.CV↗

arXiv2021

Quantum Advantage with Seeded Squeezed Light for Absorption Measurement

Absorption measurement is an exceptionally versatile tool for many applications in science and engineering. For absorption measurements using laser beams of light, the sensitivity is theoretically limited by the shot noise due to the fundamental Poisson distribution of photon number in laser radiation. In practice, the shot-noise limit can only be achieved when all other sources of noise are eliminated. \textcolor{black}{Here, we use seeded squeezed light to demonstrate that direct absorption measurement can be performed with sensitivity beyond the shot-noise limit. We present a practically realizable scheme, where intensity squeezed beams are generated by a seeded four-wave mixing process in an atomic rubidium vapor cell. More than 1.2~dB quantum advantage for the measurement sensitivity is obtained at faint absorption levels ($\leq 10\%$). We also present a detailed theoretical analysis to show that the observed quantum advantage when corrected for optical loss would be equivalent to 3~dB. Our experiment demonstrates a direct sub-shot-noise measurement of absorption that requires neither homodyne/lock-in nor logic coincidence detection schemes. It is therefore very applicable in many circumstances where sub-shot-noise-level absorption measurements are highly desirable.

physics.optics↗

arXiv2021

Drafting and Revision: Laplacian Pyramid Network for Fast High-Quality Artistic Style Transfer

Artistic style transfer aims at migrating the style from an example image to a content image. Currently, optimization-based methods have achieved great stylization quality, but expensive time cost restricts their practical applications. Meanwhile, feed-forward methods still fail to synthesize complex style, especially when holistic global and local patterns exist. Inspired by the common painting process of drawing a draft and revising the details, we introduce a novel feed-forward method named Laplacian Pyramid Network (LapStyle). LapStyle first transfers global style patterns in low-resolution via a Drafting Network. It then revises the local details in high-resolution via a Revision Network, which hallucinates a residual image according to the draft and the image textures extracted by Laplacian filtering. Higher resolution details can be easily generated by stacking Revision Networks with multiple Laplacian pyramid levels. The final stylized image is obtained by aggregating outputs of all pyramid levels. %We also introduce a patch discriminator to better learn local patterns adversarially. Experiments demonstrate that our method can synthesize high quality stylized images in real time, where holistic style patterns are properly transferred.

cs.CV↗

arXiv2021

Learning Semantic Person Image Generation by Region-Adaptive Normalization

Human pose transfer has received great attention due to its wide applications, yet is still a challenging task that is not well solved. Recent works have achieved great success to transfer the person image from the source to the target pose. However, most of them cannot well capture the semantic appearance, resulting in inconsistent and less realistic textures on the reconstructed results. To address this issue, we propose a new two-stage framework to handle the pose and appearance translation. In the first stage, we predict the target semantic parsing maps to eliminate the difficulties of pose transfer and further benefit the latter translation of per-region appearance style. In the second one, with the predicted target semantic maps, we suggest a new person image generation method by incorporating the region-adaptive normalization, in which it takes the per-region styles to guide the target appearance generation. Extensive experiments show that our proposed SPGNet can generate more semantic, consistent, and photo-realistic results and perform favorably against the state of the art methods in terms of quantitative and qualitative evaluation. The source code and model are available at https://github.com/cszy98/SPGNet.git.

cs.CV↗