arXiv subjects

Zijun Wei

Publications and source records attributed to Zijun Wei.

23 records · Page 2Linked to original sources

arXiv2023

LightPainter: Interactive Portrait Relighting with Freehand Scribble

Recent portrait relighting methods have achieved realistic results of portrait lighting effects given a desired lighting representation such as an environment map. However, these methods are not intuitive for user interaction and lack precise lighting control. We introduce LightPainter, a scribble-based relighting system that allows users to interactively manipulate portrait lighting effect with ease. This is achieved by two conditional neural networks, a delighting module that recovers geometry and albedo optionally conditioned on skin tone, and a scribble-based module for relighting. To train the relighting module, we propose a novel scribble simulation procedure to mimic real user scribbles, which allows our pipeline to be trained without any human annotations. We demonstrate high-quality and flexible portrait lighting editing capability with both quantitative and qualitative experiments. User study comparisons with commercial lighting editing tools also demonstrate consistent user preference for our method.

cs.CV↗

arXiv2022

Interactive Portrait Harmonization

Current image harmonization methods consider the entire background as the guidance for harmonization. However, this may limit the capability for user to choose any specific object/person in the background to guide the harmonization. To enable flexible interaction between user and harmonization, we introduce interactive harmonization, a new setting where the harmonization is performed with respect to a selected \emph{region} in the reference image instead of the entire background. A new flexible framework that allows users to pick certain regions of the background image and use it to guide the harmonization is proposed. Inspired by professional portrait harmonization users, we also introduce a new luminance matching loss to optimally match the color/luminance conditions between the composite foreground and select reference region. This framework provides more control to the image harmonization pipeline achieving visually pleasing portrait edits. Furthermore, we also introduce a new dataset carefully curated for validating portrait harmonization. Extensive experiments on both synthetic and real-world datasets show that the proposed approach is efficient and robust compared to previous harmonization baselines, especially for portraits. Project Webpage at \href{https://jeya-maria-jose.github.io/IPH-web/}{https://jeya-maria-jose.github.io/IPH-web/}

cs.CV↗

arXiv2021

Lite Vision Transformer with Enhanced Self-Attention

Despite the impressive representation capacity of vision transformer models, current light-weight vision transformer models still suffer from inconsistent and incorrect dense predictions at local regions. We suspect that the power of their self-attention mechanism is limited in shallower and thinner networks. We propose Lite Vision Transformer (LVT), a novel light-weight transformer network with two enhanced self-attention mechanisms to improve the model performances for mobile deployment. For the low-level features, we introduce Convolutional Self-Attention (CSA). Unlike previous approaches of merging convolution and self-attention, CSA introduces local self-attention into the convolution within a kernel of size 3x3 to enrich low-level features in the first stage of LVT. For the high-level features, we propose Recursive Atrous Self-Attention (RASA), which utilizes the multi-scale context when calculating the similarity map and a recursive mechanism to increase the representation capability with marginal extra parameter cost. The superiority of LVT is demonstrated on ImageNet recognition, ADE20K semantic segmentation, and COCO panoptic segmentation. The code is made publicly available.

cs.CV↗

arXiv2021

Structure and magnetic properties of melilite-type compounds RE2Be2GeO7 (RE = Pr, Nd, Gd-Yb) with Rare-Earth ions on Shastry-Sutherland lattice

Rare-earth (RE) based frustrated magnets as typical systems of combining strong spin-orbit coupling, geometric frustration and anisotropic exchange interactions, can give rise to diverse exotic magnetic ground states such as quantum spin liquid (QSL). The discovery of new RE-based frustrated materials is crucial for exploring the exotic magnetic phases. Herein, we report the synthesis, structure and magnetic properties of a family of melilite-type RE2Be2GeO7 (RE = Pr, Nd, Gd-Yb) compounds crystallized in a tetragonal structure, where magnetic RE3+ ions lay out on Shastry-Sutherland lattice (SSL) within ab-plane and are well separated by nonmagnetic GeBe2O7 polyhedrons along c-axis. Temperature-dependent susceptibilities and isothermal magnetization M(H) measurements reveal that most RE2Be2GeO7 compounds except RE=Tb show no magnetic ordering down to 2 K despite the dominant antiferromagnetic (AFM) interactions, where Tb2Be2GeO7 undergoes AFM transition with Neel temperature TN~ 2.5 K and field-induced spin flop behaviors (T< TN). In addition, the calculated magnetic entropy change from the isothermal M(H) curves reveal a viable magnetocaloric effect (MCE) for RE2Be2GeO7 (RE =Gd, Dy) in liquid helium temperature regimes, Gd2Be2GeO7 shows maximum Sm up to 54.8 J K-1 Kg-1 at H= 7 T and Dy2Be2GeO7 has largest value Sm=16.1 J K-1 kg-1 at H= 2 T in this family. More excitingly, rich diversity of RE ions in this family enables an archetype for exploring exotic quantum magnetic phenomena with large variability of spin located on SSL lattice.

cond-mat.mtrl-sci↗

arXiv2020

Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning

Being able to predict human gaze behavior has obvious importance for behavioral vision and for computer vision applications. Most models have mainly focused on predicting free-viewing behavior using saliency maps, but these predictions do not generalize to goal-directed behavior, such as when a person searches for a visual target object. We propose the first inverse reinforcement learning (IRL) model to learn the internal reward function and policy used by humans during visual search. The viewer's internal belief states were modeled as dynamic contextual belief maps of object locations. These maps were learned by IRL and then used to predict behavioral scanpaths for multiple target categories. To train and evaluate our IRL model we created COCO-Search18, which is now the largest dataset of high-quality search fixations in existence. COCO-Search18 has 10 participants searching for each of 18 target-object categories in 6202 images, making about 300,000 goal-directed fixations. When trained and evaluated on COCO-Search18, the IRL model outperformed baseline models in predicting search fixation scanpaths, both in terms of similarity to human search behavior and search efficiency. Finally, reward maps recovered by the IRL model reveal distinctive target-dependent patterns of object prioritization, which we interpret as a learned object context.

cs.CV↗