arXiv subjects

Jihwan Kim

Publications and source records attributed to Jihwan Kim.

At least 37 records · Page 2Linked to original sources

arXiv2024

Activating Self-Attention for Multi-Scene Absolute Pose Regression

Multi-scene absolute pose regression addresses the demand for fast and memory-efficient camera pose estimation across various real-world environments. Nowadays, transformer-based model has been devised to regress the camera pose directly in multi-scenes. Despite its potential, transformer encoders are underutilized due to the collapsed self-attention map, having low representation capacity. This work highlights the problem and investigates it from a new perspective: distortion of query-key embedding space. Based on the statistical analysis, we reveal that queries and keys are mapped in completely different spaces while only a few keys are blended into the query region. This leads to the collapse of the self-attention map as all queries are considered similar to those few keys. Therefore, we propose simple but effective solutions to activate self-attention. Concretely, we present an auxiliary loss that aligns queries and keys, preventing the distortion of query-key space and encouraging the model to find global relations by self-attention. In addition, the fixed sinusoidal positional encoding is adopted instead of undertrained learnable one to reflect appropriate positional clues into the inputs of self-attention. As a result, our approach resolves the aforementioned problem effectively, thus outperforming existing methods in both outdoor and indoor scenes.

cs.CV↗

arXiv2024

Single-shot reconstruction of three-dimensional morphology of biological cells in digital holographic microscopy using a physics-driven neural network

Recent advances in deep learning-based image reconstruction techniques have led to significant progress in phase retrieval using digital in-line holographic microscopy (DIHM). However, existing deep learning-based phase retrieval methods have technical limitations in generalization performance and three-dimensional (3D) morphology reconstruction from a single-shot hologram of biological cells. In this study, we propose a novel deep learning model, named MorpHoloNet, for single-shot reconstruction of 3D morphology by integrating physics-driven and coordinate-based neural networks. By simulating the optical diffraction of coherent light through a 3D phase shift distribution, the proposed MorpHoloNet is optimized by minimizing the loss between the simulated and input holograms on the sensor plane. Compared to existing DIHM methods that face challenges with twin image and phase retrieval problems, MorpHoloNet enables direct reconstruction of 3D complex light field and 3D morphology of a test sample from its single-shot hologram without requiring multiple phase-shifted holograms or angle scanning. The performance of the proposed MorpHoloNet is validated by reconstructing 3D morphologies and refractive index distributions from synthetic holograms of ellipsoids and experimental holograms of biological cells. The proposed deep learning model is utilized to reconstruct spatiotemporal variations in 3D translational and rotational behaviors and morphological deformations of biological cells from consecutive single-shot holograms captured using DIHM. MorpHoloNet would pave the way for advancing label-free, real-time 3D imaging and dynamic analysis of biological cells under various cellular microenvironments in biomedical and engineering fields.

cs.CV↗

arXiv2024

Prediction-Feedback DETR for Temporal Action Detection

Temporal Action Detection (TAD) is fundamental yet challenging for real-world video applications. Leveraging the unique benefits of transformers, various DETR-based approaches have been adopted in TAD. However, it has recently been identified that the attention collapse in self-attention causes the performance degradation of DETR for TAD. Building upon previous research, this paper newly addresses the attention collapse problem in cross-attention within DETR-based TAD methods. Moreover, our findings reveal that cross-attention exhibits patterns distinct from predictions, indicating a short-cut phenomenon. To resolve this, we propose a new framework, Prediction-Feedback DETR (Pred-DETR), which utilizes predictions to restore the collapse and align the cross- and self-attention with predictions. Specifically, we devise novel prediction-feedback objectives using guidance from the relations of the predictions. As a result, Pred-DETR significantly alleviates the collapse and achieves state-of-the-art performance among DETR-based methods on various challenging benchmarks including THUMOS14, ActivityNet-v1.3, HACS, and FineAction.

cs.CV↗

arXiv2024

Long-term Pre-training for Temporal Action Detection with Transformers

Temporal action detection (TAD) is challenging, yet fundamental for real-world video applications. Recently, DETR-based models for TAD have been prevailing thanks to their unique benefits. However, transformers demand a huge dataset, and unfortunately data scarcity in TAD causes a severe degeneration. In this paper, we identify two crucial problems from data scarcity: attention collapse and imbalanced performance. To this end, we propose a new pre-training strategy, Long-Term Pre-training (LTP), tailored for transformers. LTP has two main components: 1) class-wise synthesis, 2) long-term pretext tasks. Firstly, we synthesize long-form video features by merging video snippets of a target class and non-target classes. They are analogous to untrimmed data used in TAD, despite being created from trimmed data. In addition, we devise two types of long-term pretext tasks to learn long-term dependency. They impose long-term conditions such as finding second-to-fourth or short-duration actions. Our extensive experiments show state-of-the-art performances in DETR-based methods on ActivityNet-v1.3 and THUMOS14 by a large margin. Moreover, we demonstrate that LTP significantly relieves the data scarcity issues in TAD.

cs.CV↗

arXiv2024

TWLV-I: Analysis and Insights from Holistic Evaluation on Video Foundation Models

In this work, we discuss evaluating video foundation models in a fair and robust manner. Unlike language or image foundation models, many video foundation models are evaluated with differing parameters (such as sampling rate, number of frames, pretraining steps, etc.), making fair and robust comparisons challenging. Therefore, we present a carefully designed evaluation framework for measuring two core capabilities of video comprehension: appearance and motion understanding. Our findings reveal that existing video foundation models, whether text-supervised like UMT or InternVideo2, or self-supervised like V-JEPA, exhibit limitations in at least one of these capabilities. As an alternative, we introduce TWLV-I, a new video foundation model that constructs robust visual representations for both motion- and appearance-based videos. Based on the average top-1 accuracy of linear probing on five action recognition benchmarks, pretrained only on publicly accessible datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a 7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a 2.7%p improvement compared to V-JEPA (ViT-H) and a 2.8%p improvement compared to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from videos of several commonly used video benchmarks, along with evaluation source code that can directly utilize these embeddings. The code is available at https://github.com/twelvelabs-io/video-embeddings-evaluation-framework.

cs.CV↗

arXiv2024

Mutually-Aware Feature Learning for Few-Shot Object Counting

Few-shot object counting has garnered significant attention for its practicality as it aims to count target objects in a query image based on given exemplars without additional training. However, the prevailing extract-and-match approach has a shortcoming: query and exemplar features lack interaction during feature extraction since they are extracted independently and later correlated based on similarity. This can lead to insufficient target awareness and confusion in identifying the actual target when multiple class objects coexist. To address this, we propose a novel framework, Mutually-Aware FEAture learning (MAFEA), which encodes query and exemplar features with mutual awareness from the outset. By encouraging interaction throughout the pipeline, we obtain target-aware features robust to a multi-category scenario. Furthermore, we introduce background token to effectively associate the query's target region with exemplars and decouple its background region. Our extensive experiments demonstrate that our model achieves state-of-the-art performance on FSCD-LVIS and FSC-147 benchmarks with remarkably reduced target confusion.

cs.CV↗

arXiv2024

Boundary-Recovering Network for Temporal Action Detection

Temporal action detection (TAD) is challenging, yet fundamental for real-world video applications. Large temporal scale variation of actions is one of the most primary difficulties in TAD. Naturally, multi-scale features have potential in localizing actions of diverse lengths as widely used in object detection. Nevertheless, unlike objects in images, actions have more ambiguity in their boundaries. That is, small neighboring objects are not considered as a large one while short adjoining actions can be misunderstood as a long one. In the coarse-to-fine feature pyramid via pooling, these vague action boundaries can fade out, which we call 'vanishing boundary problem'. To this end, we propose Boundary-Recovering Network (BRN) to address the vanishing boundary problem. BRN constructs scale-time features by introducing a new axis called scale dimension by interpolating multi-scale features to the same temporal length. On top of scale-time features, scale-time blocks learn to exchange features across scale levels, which can effectively settle down the issue. Our extensive experiments demonstrate that our model outperforms the state-of-the-art on the two challenging benchmarks, ActivityNet-v1.3 and THUMOS14, with remarkably reduced degree of the vanishing boundary problem.

cs.CV↗

arXiv2024

Microwave Quantum Illumination with Optical Memory and Single-Mode Phase-Conjugate Receiver

Microwave quantum illumination with entangled pairs of microwave signal and optical idler modes, can achieve the sub-optimal performance with joint measurement of the signal and idler modes. Here, we first propose a testbed of microwave quantum illumination with an optical memory which is simulated with a delay line in the idler mode. It provides how much an input two-mode squeezing is necessary to compensate the loss of the optical memory, while maintaining quantum advantage over coherent state. When the memory is lossy, the input two-mode squeezing has to be higher through high cooperativity in the optical mode. Under the testbed, we propose a single-mode phase conjugate receiver that consists of a low-reflectivity beam splitter, an electro-optomechanical phase conjugator, and a photon number resolving detector. The performance of the newly proposed receiver approaches the maximum quantum advantage for local measurement. Furthermore, the quantum advantage is obtained even with an on-off detection while being robust against the loss of the memory.

quant-ph↗

arXiv2024

FIFO-Diffusion: Generating Infinite Videos from Text without Training

We propose a novel inference technique based on a pretrained diffusion model for text-conditional video generation. Our approach, called FIFO-Diffusion, is conceptually capable of generating infinitely long videos without additional training. This is achieved by iteratively performing diagonal denoising, which simultaneously processes a series of consecutive frames with increasing noise levels in a queue; our method dequeues a fully denoised frame at the head while enqueuing a new random noise frame at the tail. However, diagonal denoising is a double-edged sword as the frames near the tail can take advantage of cleaner frames by forward reference but such a strategy induces the discrepancy between training and inference. Hence, we introduce latent partitioning to reduce the training-inference gap and lookahead denoising to leverage the benefit of forward referencing. Practically, FIFO-Diffusion consumes a constant amount of memory regardless of the target video length given a baseline model, while well-suited for parallel inference on multiple GPUs. We have demonstrated the promising results and effectiveness of the proposed methods on existing text-to-video generation baselines. Generated video examples and source codes are available at our project page.

cs.CV↗

arXiv2023

Uncovering a Paleotsunami Triggered by Mass-Movement in an Alpine Lake

Mass movements and delta collapses are significant sources of tsunamis in lacustrine environments, impacting human societies enormously. Palaeotsunamis play an essential role in understanding historical events and their consequences along with their return periods. Here, we focus on a palaeo event that occurred during the Younger Dryas to Early Holocene climatic transition, ca., 12,000 years ago in the Lake Aiguebelette (NW Alps, France). Based on highresolution seismic and bathymetric surveys and sedimentological, geochemical, and magnetic analyses, a seismically induced large mass transport deposit with an initial volume of 767172 m3 was identified, dated and mapped. To investigate whether this underwater mass transport produced a palaeotsunami in the Lake Aiguebelette, this research combines sedimentary records and numerical models. Numerical simulations of tsunamis are performed using a viscoplastic landslide model for tsunami source generation and two-dimensional depth-averaged nonlinear shallow water equations for tsunami wave propagation and inundation modelling. Our simulations conclude that this sublacustrine landslide produced a tsunami wave with a maximum amplitude of approximately 2 m and run-up heights of up to 3.6 m. The modelled sediment thickness resulting from this mass transport corroborates well with the event deposits mapped in the lake. Based on our results, we suggest that this sublacustrine mass transport generated a significant tsunami wave that has not been reported previously to the best of our knowledge.

math.NA↗

arXiv2023

Self-Feedback DETR for Temporal Action Detection

Temporal Action Detection (TAD) is challenging but fundamental for real-world video applications. Recently, DETR-based models have been devised for TAD but have not performed well yet. In this paper, we point out the problem in the self-attention of DETR for TAD; the attention modules focus on a few key elements, called temporal collapse problem. It degrades the capability of the encoder and decoder since their self-attention modules play no role. To solve the problem, we propose a novel framework, Self-DETR, which utilizes cross-attention maps of the decoder to reactivate self-attention modules. We recover the relationship between encoder features by simple matrix multiplication of the cross-attention map and its transpose. Likewise, we also get the information within decoder queries. By guiding collapsed self-attention maps with the guidance map calculated, we settle down the temporal collapse of self-attention modules in the encoder and decoder. Our extensive experiments demonstrate that Self-DETR resolves the temporal collapse problem by keeping high diversity of attention over all layers.

cs.CV↗

arXiv2023

When Crowd Meets Persona: Creating a Large-Scale Open-Domain Persona Dialogue Corpus

Building a natural language dataset requires caution since word semantics is vulnerable to subtle text change or the definition of the annotated concept. Such a tendency can be seen in generative tasks like question-answering and dialogue generation and also in tasks that create a categorization-based corpus, like topic classification or sentiment analysis. Open-domain conversations involve two or more crowdworkers freely conversing about any topic, and collecting such data is particularly difficult for two reasons: 1) the dataset should be ``crafted" rather than ``obtained" due to privacy concerns, and 2) paid creation of such dialogues may differ from how crowdworkers behave in real-world settings. In this study, we tackle these issues when creating a large-scale open-domain persona dialogue corpus, where persona implies that the conversation is performed by several actors with a fixed persona and user-side workers from an unspecified crowd.

cs.CL↗

arXiv2023

Gaussian Quantum Illumination via Monotone Metrics

Quantum illumination is to discern the presence or absence of a low reflectivity target, where the error probability decays exponentially in the number of copies used. When the target reflectivity is small so that it is hard to distinguish target presence or absence, the exponential decay constant falls into a class of objects called monotone metrics. We evaluate monotone metrics restricted to Gaussian states in terms of first-order moments and covariance matrix. Under the assumption of a low reflectivity target, we explicitly derive analytic formulae for decay constant of an arbitrary Gaussian input state. Especially, in the limit of large background noise and low reflectivity, there is no need of symplectic diagonalization which usually complicates the computation of decay constants. First, we show that two-mode squeezed vacuum (TMSV) states are the optimal probe among pure Gaussian states with fixed signal mean photon number. Second, as an alternative to preparing TMSV states with high mean photon number, we show that preparing a TMSV state with low mean photon number and displacing the signal mode is a more experimentally feasible setup without degrading the performance that much. Third, we show that it is of utmost importance to prepare an efficient idler memory to beat coherent states and provide analytic bounds on the idler memory transmittivity in terms of signal power, background noise, and idler memory noise. Finally, we identify the region of physically possible correlations between the signal and idler modes that can beat coherent states.

quant-ph↗

arXiv2022

Janus van der Waals equations for real molecules with two-sided phase transitions

We obtain families of generalised van der Waals equations characterised by an even number $n=2,4,6$ and a continuous free parameter which is tunable for a critical compressibility factor. Each equation features two adjacent critical points which have a common critical temperature $T_{c}$ and arbitrarily close two critical densities. The critical phase transitions are naturally two-sided: the critical exponents are $\alpha_{\scriptscriptstyle{P}}=\gamma_{\scriptscriptstyle{P}}=\frac{2}{3}$, $\beta_{\scriptscriptstyle{P}}=\delta^{-1}=\frac{1}{3}$ for $T>T_{c}$ and $\alpha_{\scriptscriptstyle{P}}=\gamma_{\scriptscriptstyle{P}}=\frac{n}{n+1}$, $\beta_{\scriptscriptstyle{P}}=\delta^{-1}=\frac{1}{n+1}$ for $T<T_{c}$. In contrast with the original van der Waals equation, our novel equations all reduce consistently to the classical ideal gas law in low density limit. We test our formulas against NIST data for eleven major molecules and show agreements better than the original van der Waals equation, not only near to the critical points but also in low density regions.

physics.chem-ph↗

arXiv2021

Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification

In this paper, we propose self-supervised speaker representation learning strategies, which comprise of a bootstrap equilibrium speaker representation learning in the front-end and an uncertainty-aware probabilistic speaker embedding training in the back-end. In the front-end stage, we learn the speaker representations via the bootstrap training scheme with the uniformity regularization term. In the back-end stage, the probabilistic speaker embeddings are estimated by maximizing the mutual likelihood score between the speech samples belonging to the same speaker, which provide not only speaker representations but also data uncertainty. Experimental results show that the proposed bootstrap equilibrium training strategy can effectively help learn the speaker representations and outperforms the conventional methods based on contrastive learning. Also, we demonstrate that the integrated two-stage framework further improves the speaker verification performance on the VoxCeleb1 test set in terms of EER and MinDCF.

eess.AS↗

arXiv2020

Superconducting Nanoelectromechanical Transducer Resilient to Magnetic Fields

Nanoscale electromechanical coupling provides a unique route towards control of mechanical motions and microwave fields in superconducting cavity electromechanical devices. Though their successes in utilizing the optomechanical or electromechanical back-action effects for various purposes, aluminum imposes severe constraints on their operating conditions with its low superconducting critical temperature (1.2 K) and magnetic field (0.01 T). To extend the potential of the devices, here we fabricate a superconducting electromechanical device employing niobium and demonstrate a set of cavity electromechanical dynamics including back-action cooling and amplification, and electromechanically induced reflection at 4.2 K and in strong magnetic fields up to 0.8 T. This device could be used to realize electromechanical microwave components for quantum technologies by integrating amplifiers, converters, and circulators on a single chip that can be installed at the 4K stage of dilution refrigerators. Moreover, with its ability to control and readout nanomechanical motions simultaneously, this niobium electromechanical transducer could provide powerful nanomechanical sensing platforms.

physics.app-ph↗

arXiv2019

Exploring Lexicon-Free Modeling Units for End-to-End Korean and Korean-English Code-Switching Speech Recognition

As the character-based end-to-end automatic speech recognition (ASR) models evolve, the choice of acoustic modeling units becomes important. Since Korean is a fairly phonetic language and has a unique writing system with its own Korean alphabet, it's worth investigating modeling units for an end-to-end Korean ASR task. In this work, we introduce lexicon-free modeling units in Korean, and explore them using a hybrid CTC/Attention-based encoder-decoder model. Five lexicon-free units are investigated: Syllable-based Korean character (with English character for a code-switching task), Korean Jamo character (with English character), sub-word on syllable-based character (with sub-word in English), sub-word on Jamo character (with sub-words in English), and finally byte unit, which is a universal one across language. Experiments on Zeroth-Korean (51.6 hrs) and Medical Record (2530 hrs) are done for Korean and Korean-English code-switching ASR tasks, respectively. Sequence-to-sequence learning with sub-words based on Korean syllables (and sub-words in English) performs the best for both tasks without lexicon and an extra language model integration.

cs.SD↗

arXiv2019

Nanomechanical characterization of quantum interference in a topological insulator nanowire

The discovery of two-dimensional gapless Dirac fermions in graphene and topological insulators (TI) has sparked extensive ongoing research toward applications of their unique electronic properties. The gapless surface states in three-dimensional insulators indicate a distinct topological phase of matter with a non-trivial Z2 invariant that can be verified by angle-resolved photoemission spectroscopy or magnetoresistance quantum oscillation. In TI nanowires, the gapless surface states exhibit Aharonov-Bohm (AB) oscillations in conductance, with this quantum interference effect accompanying a change in the number of transverse one-dimensional modes in transport. Thus, while the density of states (DOS) of such nanowires is expected to show such AB oscillation, this effect has yet to be observed. Here, we adopt nanomechanical measurements that reveal AB oscillations in the DOS of a topological insulator. The TI nanowire under study is an electromechanical resonator embedded in an electrical circuit, and quantum capacitance effects from DOS oscillation modulate the circuit capacitance thereby altering the spring constant to generate mechanical resonant frequency shifts. Detection of the quantum capacitance effects from surface-state DOS is facilitated by the small effective capacitances and high quality factors of nanomechanical resonators, and as such the present technique could be extended to study diverse quantum materials at nanoscale.

cond-mat.mes-hall↗