arXiv subjects

Beomyoung Kim

Publications and source records attributed to Beomyoung Kim.

At least 19 recordsLinked to original sources

arXiv2026

Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation

Despite significant advances in image segmentation, even state-of-the-art models produce masks with imperfect boundaries, semantic inconsistencies, and structural errors. Mask refinement addresses these limitations, yet current approaches rely on simplistic synthetic noise that fails to capture the complex error patterns of real segmentation models. We introduce Phoenix, a novel framework that leverages adversarial learning to generate semantically meaningful noise patterns and contrastive learning to model refinement relationships. Our approach consists of two key innovations: (1) Adversarial Mask Perturbation, which employs embedding attacks to create semantic-aware noise that mimics real segmentation errors, and (2) Contrastive Mask Refinement Learning, which establishes a tri-directional framework that ensures feature consistency within semantic regions while maintaining separation between classes. Experiments demonstrate that Phoenix significantly outperforms existing methods across diverse tasks, while consistently enhancing state-of-the-art segmentation models with substantial improvements. Our code and project page are publicly available at https://phoenix-eccv26.github.io.

cs.CV↗

arXiv2024

ZIM: Zero-Shot Image Matting for Anything

The recent segmentation foundation model, Segment Anything Model (SAM), exhibits strong zero-shot segmentation capabilities, but it falls short in generating fine-grained precise masks. To address this limitation, we propose a novel zero-shot image matting model, called ZIM, with two key contributions: First, we develop a label converter that transforms segmentation labels into detailed matte labels, constructing the new SA1B-Matte dataset without costly manual annotations. Training SAM with this dataset enables it to generate precise matte masks while maintaining its zero-shot capability. Second, we design the zero-shot matting model equipped with a hierarchical pixel decoder to enhance mask representation, along with a prompt-aware masked attention mechanism to improve performance by enabling the model to focus on regions specified by visual prompts. We evaluate ZIM using the newly introduced MicroMat-3K test set, which contains high-quality micro-level matte labels. Experimental results show that ZIM outperforms existing methods in fine-grained mask generation and zero-shot generalization. Furthermore, we demonstrate the versatility of ZIM in various downstream tasks requiring precise masks, such as image inpainting and 3D NeRF. Our contributions provide a robust foundation for advancing zero-shot matting and its downstream applications across a wide range of computer vision tasks. The code is available at https://github.com/naver-ai/ZIM.

cs.CV↗

arXiv2024

Rethinking Saliency-Guided Weakly-Supervised Semantic Segmentation

This paper presents a fresh perspective on the role of saliency maps in weakly-supervised semantic segmentation (WSSS) and offers new insights and research directions based on our empirical findings. We conduct comprehensive experiments and observe that the quality of the saliency map is a critical factor in saliency-guided WSSS approaches. Nonetheless, we find that the saliency maps used in previous works are often arbitrarily chosen, despite their significant impact on WSSS. Additionally, we observe that the choice of the threshold, which has received less attention before, is non-trivial in WSSS. To facilitate more meaningful and rigorous research for saliency-guided WSSS, we introduce \texttt{WSSS-BED}, a standardized framework for conducting research under unified conditions. \texttt{WSSS-BED} provides various saliency maps and activation maps for seven WSSS methods, as well as saliency maps from unsupervised salient object detection models.

cs.CV↗

arXiv2024

Towards Label-Efficient Human Matting: A Simple Baseline for Weakly Semi-Supervised Trimap-Free Human Matting

This paper presents a new practical training method for human matting, which demands delicate pixel-level human region identification and significantly laborious annotations. To reduce the annotation cost, most existing matting approaches often rely on image synthesis to augment the dataset. However, the unnaturalness of synthesized training images brings in a new domain generalization challenge for natural images. To address this challenge, we introduce a new learning paradigm, weakly semi-supervised human matting (WSSHM), which leverages a small amount of expensive matte labels and a large amount of budget-friendly segmentation labels, to save the annotation cost and resolve the domain generalization problem. To achieve the goal of WSSHM, we propose a simple and effective training method, named Matte Label Blending (MLB), that selectively guides only the beneficial knowledge of the segmentation and matte data to the matting model. Extensive experiments with our detailed analysis demonstrate our method can substantially improve the robustness of the matting model using a few matte data and numerous segmentation data. Our training method is also easily applicable to real-time models, achieving competitive accuracy with breakneck inference speed (328 FPS on NVIDIA V100 GPU). The implementation code is available at \url{https://github.com/clovaai/WSSHM}.

cs.CV↗

arXiv2024

ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning

Panoptic segmentation, combining semantic and instance segmentation, stands as a cutting-edge computer vision task. Despite recent progress with deep learning models, the dynamic nature of real-world applications necessitates continual learning, where models adapt to new classes (plasticity) over time without forgetting old ones (catastrophic forgetting). Current continual segmentation methods often rely on distillation strategies like knowledge distillation and pseudo-labeling, which are effective but result in increased training complexity and computational overhead. In this paper, we introduce a novel and efficient method for continual panoptic segmentation based on Visual Prompt Tuning, dubbed ECLIPSE. Our approach involves freezing the base model parameters and fine-tuning only a small set of prompt embeddings, addressing both catastrophic forgetting and plasticity and significantly reducing the trainable parameters. To mitigate inherent challenges such as error propagation and semantic drift in continual segmentation, we propose logit manipulation to effectively leverage common knowledge across the classes. Experiments on ADE20K continual panoptic segmentation benchmark demonstrate the superiority of ECLIPSE, notably its robustness against catastrophic forgetting and its reasonable plasticity, achieving a new state-of-the-art. The code is available at https://github.com/clovaai/ECLIPSE.

cs.CV↗

arXiv2023

The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation

In this paper, we introduce a novel learning scheme named weakly semi-supervised instance segmentation (WSSIS) with point labels for budget-efficient and high-performance instance segmentation. Namely, we consider a dataset setting consisting of a few fully-labeled images and a lot of point-labeled images. Motivated by the main challenge of semi-supervised approaches mainly derives from the trade-off between false-negative and false-positive instance proposals, we propose a method for WSSIS that can effectively leverage the budget-friendly point labels as a powerful weak supervision source to resolve the challenge. Furthermore, to deal with the hard case where the amount of fully-labeled data is extremely limited, we propose a MaskRefineNet that refines noise in rough masks. We conduct extensive experiments on COCO and BDD100K datasets, and the proposed method achieves promising results comparable to those of the fully-supervised model, even with 50% of the fully labeled COCO data (38.8% vs. 39.7%). Moreover, when using as little as 5% of fully labeled COCO data, our method shows significantly superior performance over the state-of-the-art semi-supervised learning method (33.7% vs. 24.9%). The code is available at https://github.com/clovaai/PointWSSIS.

cs.CV↗

arXiv2022

EResFD: Rediscovery of the Effectiveness of Standard Convolution for Lightweight Face Detection

This paper analyzes the design choices of face detection architecture that improve efficiency of computation cost and accuracy. Specifically, we re-examine the effectiveness of the standard convolutional block as a lightweight backbone architecture for face detection. Unlike the current tendency of lightweight architecture design, which heavily utilizes depthwise separable convolution layers, we show that heavily channel-pruned standard convolution layers can achieve better accuracy and inference speed when using a similar parameter size. This observation is supported by the analyses concerning the characteristics of the target data domain, faces. Based on our observation, we propose to employ ResNet with a highly reduced channel, which surprisingly allows high efficiency compared to other mobile-friendly networks (e.g., MobileNetV1, V2, V3). From the extensive experiments, we show that the proposed backbone can replace that of the state-of-the-art face detector with a faster inference speed. Also, we further propose a new feature aggregation method to maximize the detection performance. Our proposed detector EResFD obtained 80.4% mAP on WIDER FACE Hard subset which only takes 37.7 ms for VGA image inference on CPU. Code is available at https://github.com/clovaai/EResFD.

cs.CV↗

arXiv2022

Learning Features with Parameter-Free Layers

Trainable layers such as convolutional building blocks are the standard network design choices by learning parameters to capture the global context through successive spatial operations. When designing an efficient network, trainable layers such as the depthwise convolution is the source of efficiency in the number of parameters and FLOPs, but there was little improvement to the model speed in practice. This paper argues that simple built-in parameter-free operations can be a favorable alternative to the efficient trainable layers replacing spatial operations in a network architecture. We aim to break the stereotype of organizing the spatial operations of building blocks into trainable layers. Extensive experimental analyses based on layer-level studies with fully-trained models and neural architecture searches are provided to investigate whether parameter-free operations such as the max-pool are functional. The studies eventually give us a simple yet effective idea for redesigning network architectures, where the parameter-free operations are heavily used as the main building block without sacrificing the model accuracy as much. Experimental results on the ImageNet dataset demonstrate that the network architectures with parameter-free operations could enjoy the advantages of further efficiency in terms of model speed, the number of the parameters, and FLOPs. Code and ImageNet pretrained models are available at https://github.com/naver-ai/PfLayer.

cs.CV↗

arXiv2021

Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinement

Weakly-supervised instance segmentation (WSIS) has been considered as a more challenging task than weakly-supervised semantic segmentation (WSSS). Compared to WSSS, WSIS requires instance-wise localization, which is difficult to extract from image-level labels. To tackle the problem, most WSIS approaches use off-the-shelf proposal techniques that require pre-training with instance or object level labels, deviating the fundamental definition of the fully-image-level supervised setting. In this paper, we propose a novel approach including two innovative components. First, we propose a semantic knowledge transfer to obtain pseudo instance labels by transferring the knowledge of WSSS to WSIS while eliminating the need for the off-the-shelf proposals. Second, we propose a self-refinement method to refine the pseudo instance labels in a self-supervised scheme and to use the refined labels for training in an online manner. Here, we discover an erroneous phenomenon, semantic drift, that occurred by the missing instances in pseudo instance labels categorized as background class. This semantic drift occurs confusion between background and instance in training and consequently degrades the segmentation performance. We term this problem as semantic drift problem and show that our proposed self-refinement method eliminates the semantic drift problem. The extensive experiments on PASCAL VOC 2012 and MS COCO demonstrate the effectiveness of our approach, and we achieve a considerable performance without off-the-shelf proposal techniques. The code is available at https://github.com/clovaai/BESTIE.

cs.CV↗

arXiv2021

SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning

This paper introduces a solid state-of-the-art baseline for a class-incremental semantic segmentation (CISS) problem. While the recent CISS algorithms utilize variants of the knowledge distillation (KD) technique to tackle the problem, they failed to fully address the critical challenges in CISS causing the catastrophic forgetting; the semantic drift of the background class and the multi-label prediction issue. To better address these challenges, we propose a new method, dubbed SSUL-M (Semantic Segmentation with Unknown Label with Memory), by carefully combining techniques tailored for semantic segmentation. Specifically, we claim three main contributions. (1) defining unknown classes within the background class to help to learn future classes (help plasticity), (2) freezing backbone network and past classifiers with binary cross-entropy loss and pseudo-labeling to overcome catastrophic forgetting (help stability), and (3) utilizing tiny exemplar memory for the first time in CISS to improve both plasticity and stability. The extensively conducted experiments show the effectiveness of our method, achieving significantly better performance than the recent state-of-the-art baselines on the standard benchmark datasets. Furthermore, we justify our contributions with thorough ablation analyses and discuss different natures of the CISS problem compared to the traditional class-incremental learning targeting classification. The official code is available at https://github.com/clovaai/SSUL.

cs.CV↗

arXiv2021

TricubeNet: 2D Kernel-Based Object Representation for Weakly-Occluded Oriented Object Detection

We present a novel approach for oriented object detection, named TricubeNet, which localizes oriented objects using visual cues ($i.e.,$ heatmap) instead of oriented box offsets regression. We represent each object as a 2D Tricube kernel and extract bounding boxes using simple image-processing algorithms. Our approach is able to (1) obtain well-arranged boxes from visual cues, (2) solve the angle discontinuity problem, and (3) can save computational complexity due to our anchor-free modeling. To further boost the performance, we propose some effective techniques for size-invariant loss, reducing false detections, extracting rotation-invariant features, and heatmap refinement. To demonstrate the effectiveness of our TricubeNet, we experiment on various tasks for weakly-occluded oriented object detection: detection in an aerial image, densely packed object image, and text image. The extensive experimental results show that our TricubeNet is quite effective for oriented object detection. Code is available at https://github.com/qjadud1994/TricubeNet.

cs.CV↗

arXiv2021

Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation

Weakly-supervised semantic segmentation (WSSS) using image-level labels has recently attracted much attention for reducing annotation costs. Existing WSSS methods utilize localization maps from the classification network to generate pseudo segmentation labels. However, since localization maps obtained from the classifier focus only on sparse discriminative object regions, it is difficult to generate high-quality segmentation labels. To address this issue, we introduce discriminative region suppression (DRS) module that is a simple yet effective method to expand object activation regions. DRS suppresses the attention on discriminative regions and spreads it to adjacent non-discriminative regions, generating dense localization maps. DRS requires few or no additional parameters and can be plugged into any network. Furthermore, we introduce an additional learning strategy to give a self-enhancement of localization maps, named localization map refinement learning. Benefiting from this refinement learning, localization maps are refined and enhanced by recovering some missing parts or removing noise itself. Due to its simplicity and effectiveness, our approach achieves mIoU 71.4% on the PASCAL VOC 2012 segmentation benchmark using only image-level labels. Extensive experiments demonstrate the effectiveness of our approach. The code is available at https://github.com/qjadud1994/DRS.

cs.CV↗

arXiv2020

Electric field driven octahedral rotation in perovskite

Rotation of MO6 (M = transition metal) octahedra is a key determinant of the physical properties of perovskite materials. Therefore, tuning physical properties, one of the most important goals in condensed matter research, may be accomplished by controlling octahedral rotation (OR). In this study, it is demonstrated that OR can be driven by an electric field in Sr$_2$RuO$_4$. Rotated octahedra in the surface layer of Sr$_2$RuO$_4$ are restored to the unrotated bulk structure upon dosing the surface with K. Theoretical investigation shows that OR in Sr$_2$RuO$_4$ originates from the surface electric field, which can be tuned via the screening effect of the overlaid K layer. This work establishes not only that variation in the OR angle can be induced by an electric field, but also provides a way to control OR, which is an important step towards in situ control of the physical properties of perovskite oxides.

cond-mat.mtrl-sci↗

arXiv2017

Universal Mechanism of Band-Gap Engineering in Transition-Metal Dichalcogenides

Two-dimensional (2D) van-der-Waals semiconductors have emerged as a class of materials with promising device characteristics owing to the intrinsic bandgap. For realistic applications, the ideal is to modify the bandgap in a controlled manner by a mechanism that can be generally applied to this class of materials. Here, we report the observation of a universally tunable bandgap in the family of bulk 2H transition metal dichalcogenides (TMDs) by in situ surface doping of Rb atoms. A series of angle-resolved photoemission spectra unexceptionally shows that the bandgap of TMDs at the zone corners is modulated in the range of 0.8 ~ 2.0 eV, which covers a wide spectral range from visible to near infrared, with a tendency from indirect to direct bandgap. A key clue to understand the mechanism of this bandgap engineering is provided by the spectroscopic signature of symmetry breaking and resultant spin splitting, which can be explained by the formation of 2D electric dipole layers within the surface bilayer of TMDs. Our results establish the surface Stark effect as a universal mechanism of bandgap engineering based on the strong 2D nature of van-der-Waals semiconductors.

cond-mat.mtrl-sci↗

arXiv2017

$d_{xz/yz}$ Orbital Subband Structures and Chiral Orbital Angular Momentum in the (001) Surface States of SrTiO$_3$

We have performed angle resolved photoemission spectroscopy (ARPES) experiments on the surface states of SrTiO$_3$(001) using linearly and circularly polarized light to investigate the subband structures of out-of-plane $d_{xz/yz}$ orbitals and chiral orbital angular momentum (OAM). The data taken in the first Brillouin zone reveal new subbands for $d_{xz/yz}$ orbitals with Fermi wave vectors of 0.25 and 0.45 $\mathrmÅ^{-1}$ in addition to the previously reported ones. As a result, there are at least two subbands for all the Ti 3d t$_{2g}$ orbitals. Our circular dichroism ARPES data is suggestive of a chiral OAM structure in the surface states and may provide clues to the origin of the linear Rashba-like surface band splitting.

cond-mat.mtrl-sci↗

arXiv2016

Experimental determination of the massive Dirac fermion model parameters for MoS$_2$, MoSe$_2$, WS$_2$, and WSe$_2$

Monolayer MX$_2$ (M = Mo, W; X = S, Se) has drawn much attention recently for its possible application possibilities for optoelectronics, spintronics, and valleytronics. Its exotic optical and electronic properties include a direct band gap, circular polarization dependent optical transitions, and valence band (VB) spin band splitting at the $K$ and $-K$ points. These properties can be described within a minimal model, called the massive Dirac fermion model for which the parameters need to be experimentally determined. We propose that the parameters can be obtained from angle resolved photoemission (ARPES) data from bulk 2H-MX$_2$, instead of monolayer MX$_2$. Through tight binding calculations, we show how the electronic structure at high symmetry points evolves as the system changes from the monolayer to the three dimensional bulk 2H-MX$_2$ . We find vanishing $k_z$ dispersion and almost no change in the direct band gap at the $K$ and $-K$ points, in sharp contrast to the strong $k_z$ dispersion at the $Γ$ point. These facts allow us to extract the gap and spin band splitting at the $K$ point as well as the hopping energy from bulk ARPES data. We performed ARPES experiments on single crystals of MoS$_2$, MoSe$_2$, WS$_2$, and WSe$_2$ at various photon energies and also with potassium evaporation. From the data, we determined the parameters for the massive Dirac fermion model for monolayer MoS$_2$, MoSe$_2$, WS$_2$, and WSe$_2$.

cond-mat.mtrl-sci↗

arXiv2013

Microscopic mechanism for asymmetric charge distribution in Rashba-type surface states and the origin of the the energy splitting scale

Microscopic mechanism for the Rashba-type band splitting is examined in detail. We show how asymmetric charge distribution is formed when local orbital angular momentum (OAM) and crystal momentum get interlocked due to surface effects. An electrostatic energy term in the Hamiltonian appears when such OAM and crystal momentum dependent asymmetric charge distribution is placed in an electric field produced from an inversion symmetry breaking (ISB). Analysis by using an effective Hamiltonian shows that, as the atomic spin-orbit coupling (SOC) strength increases from weak to strong, originally OAM-quenched states evolve into well-defined chiral OAM states and then to total angular momentum J-states. In addition, the energy scale of the band splitting changes from atomic SOC energy to electrostatic energy. To confirm the validity of the model, we study OAM and spin structures of Au(111) system by using an effective Hamiltonian for the d-orbitals case. As for strong SOC regime, we choose Bi2Te2Se as a prototype system. We performed circular dichroism angle resolved photoemission spectroscopy experiments as well as first-principles calculations. We find that the effective model can explain various aspects of spin and OAM structures of the system.

cond-mat.mtrl-sci↗

arXiv2011

Observation of warping effects in the band and angular momentum structures of topological insulator Bi2Te3

We performed angle resolved photoemission (ARPES) experiments on Bi2Te3 with circularly polarized light. ARPES data show very strong circular dichroism, indicating existence of orbital angular momentum (OAM). Moreover, the alignment of OAM is found to have a strong binding energy dependence. Such energy dependence comes from a relatively strong band warping effect in Bi2Te3 compared to Bi2Se3. OAM close to Dirac point has an ideal chiral structure (sin ?) without out-of-plane component. Warping effect comes in as the binding energy decreases and circular dichroism along a constant energy contour can no longer be explained by a simple sin? function but requires a sin3? term. When the warping effect becomes even stronger near the Fermi energy, circular dichroism gains an additional sin6? term. Such behavior is found to be compatible with the theoretically predicted OAM structure.

cond-mat.mtrl-sci↗