arXiv subjects

Seungju Han

Publications and source records attributed to Seungju Han.

At least 37 records · Page 2Linked to original sources

arXiv2024

SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models

Despite the recent advances of the artificial intelligence, building social intelligence remains a challenge. Among social signals, laughter is one of the distinctive expressions that occurs during social interactions between humans. In this work, we tackle a new challenge for machines to understand the rationale behind laughter in video, Video Laugh Reasoning. We introduce this new task to explain why people laugh in a particular video and a dataset for this task. Our proposed dataset, SMILE, comprises video clips and language descriptions of why people laugh. We propose a baseline by leveraging the reasoning capacity of large language models (LLMs) with textual video representation. Experiments show that our baseline can generate plausible explanations for laughter. We further investigate the scalability of our baseline by probing other video understanding tasks and in-the-wild videos. We release our dataset, code, and model checkpoints on https://github.com/postech-ami/SMILE-Dataset.

cs.CL↗

arXiv2024

Precursors to Topological Phase Transition in Topological Ladders

We study the coupling of two topologcal subsystems in distinct topological states, and show that it leads to a precursor behavior of the topological phase transition in the overall system. This behavior is solely determined by the symmetry classes of the subsystem Hamiltonians and coupling terms, and is marked by the persistent existence of subgap states within the bulk energy gap. By investigating the critical current of Josephson junctions involving topological superconductors, we also illustrate that such subgap states play crucial roles in physical properties of nanoscale devices or materials.

cond-mat.mes-hall↗

arXiv2023

Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms

Commonsense norms are defeasible by context: reading books is usually great, but not when driving a car. While contexts can be explicitly described in language, in embodied scenarios, contexts are often provided visually. This type of visually grounded reasoning about defeasible commonsense norms is generally easy for humans, but (as we show) poses a challenge for machines, as it necessitates both visual understanding and reasoning about commonsense norms. We construct a new multimodal benchmark for studying visual-grounded commonsense norms: NORMLENS. NORMLENS consists of 10K human judgments accompanied by free-form explanations covering 2K multimodal situations, and serves as a probe to address two questions: (1) to what extent can models align with average human judgment? and (2) how well can models explain their predicted judgments? We find that state-of-the-art model judgments and explanations are not well-aligned with human annotation. Additionally, we present a new approach to better align models with humans by distilling social commonsense knowledge from large language models. The data and code are released at https://seungjuhan.me/normlens.

cs.LG↗

arXiv2023

CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos

Visual information is central to conversation: body gestures and physical behaviour, for example, contribute to meaning that transcends words alone. To date, however, most neural conversational models are limited to just text. We introduce CHAMPAGNE, a generative model of conversations that can account for visual contexts. To train CHAMPAGNE, we collect and release YTD-18M, a large-scale corpus of 18M video-based dialogues. YTD-18M is constructed from web videos: crucial to our data collection pipeline is a pretrained language model that converts error-prone automatic transcripts to a cleaner dialogue format while maintaining meaning. Human evaluation reveals that YTD-18M is more sensible and specific than prior resources (MMDialog, 1M dialogues), while maintaining visual-groundedness. Experiments demonstrate that 1) CHAMPAGNE learns to conduct conversation from YTD-18M; and 2) when fine-tuned, it achieves state-of-the-art results on four vision-language tasks focused on real-world conversations. We release data, models, and code.

cs.CL↗

arXiv2022

Measuring and Improving Semantic Diversity of Dialogue Generation

Response diversity has become an important criterion for evaluating the quality of open-domain dialogue generation models. However, current evaluation metrics for response diversity often fail to capture the semantic diversity of generated responses, as they mainly consider lexical aspects of the generated responses. In this paper, we introduce a new automatic evaluation metric to measure the semantic diversity of generated responses. Through human evaluation, we demonstrate that our proposed metric captures human judgments on response diversity better than existing lexical-level diversity metrics. Furthermore, motivated by analyzing an existing dialogue dataset, we propose a simple yet effective learning method that improves the semantic diversity of generated responses. Our learning method weights training samples based on the semantic distribution of the training set. We show that our learning method improves response diversity and coherency better than other baseline methods through automatic and human evaluation.

cs.CL↗

arXiv2022

Subgap modes in two-dimensional magnetic Josephson junctions

We consider two-dimensional superconductor/ferromagnet/superconductor junctions and investigate the subgap modes along the junction interface. The subgap modes exhibit characteristics similar to the Yu-Shiba-Rusinov states that originate form the interplay between superconductivity and ferromagnetism in the magnetic junction. The dispersion relation of the subgap modes shows qualitatively different profiles depending on the transport state (metallic, half-metallic, or insulating) of the ferromagnet. As the spin splitting in the ferromagnet is increased, the subgap modes bring about a $0$-$π$ transition in the Josephson current across the junction, with the Josephson current density depending strongly on the momentum along the junction interface (i.e., the direction of the incident current). For clean superconductor-ferromagnet interfaces (i.e., strong coupling between superconductors and ferromagnet), the subgap modes develop flat quasi-particle bands that allow to engineer the wave functions of the subgap modes along an inhomogeneous magnetic junction.

cond-mat.supr-con↗

arXiv2022

Towards Accurate Facial Landmark Detection via Cascaded Transformers

Accurate facial landmarks are essential prerequisites for many tasks related to human faces. In this paper, an accurate facial landmark detector is proposed based on cascaded transformers. We formulate facial landmark detection as a coordinate regression task such that the model can be trained end-to-end. With self-attention in transformers, our model can inherently exploit the structured relationships between landmarks, which would benefit landmark detection under challenging conditions such as large pose and occlusion. During cascaded refinement, our model is able to extract the most relevant image features around the target landmark for coordinate prediction, based on deformable attention mechanism, thus bringing more accurate alignment. In addition, we propose a novel decoder that refines image features and landmark positions simultaneously. With few parameter increasing, the detection performance improves further. Our model achieves new state-of-the-art performance on several standard facial landmark detection benchmarks, and shows good generalization ability in cross-dataset evaluation.

cs.CV↗

arXiv2022

Pushing the Performance Limit of Scene Text Recognizer without Human Annotation

Scene text recognition (STR) attracts much attention over the years because of its wide application. Most methods train STR model in a fully supervised manner which requires large amounts of labeled data. Although synthetic data contributes a lot to STR, it suffers from the real-tosynthetic domain gap that restricts model performance. In this work, we aim to boost STR models by leveraging both synthetic data and the numerous real unlabeled images, exempting human annotation cost thoroughly. A robust consistency regularization based semi-supervised framework is proposed for STR, which can effectively solve the instability issue due to domain inconsistency between synthetic and real images. A character-level consistency regularization is designed to mitigate the misalignment between characters in sequence recognition. Extensive experiments on standard text recognition benchmarks demonstrate the effectiveness of the proposed method. It can steadily improve existing STR models, and boost an STR model to achieve new state-of-the-art results. To our best knowledge, this is the first consistency regularization based framework that applies successfully to STR.

cs.CV↗

arXiv2022

Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances

In this paper, we consider mimicking fictional characters as a promising direction for building engaging conversation models. To this end, we present a new practical task where only a few utterances of each fictional character are available to generate responses mimicking them. Furthermore, we propose a new method named Pseudo Dialog Prompting (PDP) that generates responses by leveraging the power of large-scale language models with prompts containing the target character's utterances. To better reflect the style of the character, PDP builds the prompts in the form of dialog that includes the character's utterances as dialog history. Since only utterances of the characters are available in the proposed task, PDP matches each utterance with an appropriate pseudo-context from a predefined set of context candidates using a retrieval model. Through human and automatic evaluation, we show that PDP generates responses that better reflect the style of fictional characters than baseline methods.

cs.CL↗

arXiv2021

Understanding and Improving the Exemplar-based Generation for Open-domain Conversation

Exemplar-based generative models for open-domain conversation produce responses based on the exemplars provided by the retriever, taking advantage of generative models and retrieval models. However, they often ignore the retrieved exemplars while generating responses or produce responses over-fitted to the retrieved exemplars. In this paper, we argue that these drawbacks are derived from the one-to-many problem of the open-domain conversation. When the retrieved exemplar is relevant to the given context yet significantly different from the gold response, the exemplar-based generative models are trained to ignore the exemplar since the exemplar is not helpful for generating the gold response. On the other hand, when the retrieved exemplar is lexically similar to the gold response, the generative models are trained to rely on the exemplar highly. Therefore, we propose a training method selecting exemplars that are semantically relevant to the gold response but lexically distanced from the gold response to mitigate the above disadvantages. In the training phase, our proposed training method first uses the gold response instead of dialogue context as a query to select exemplars that are semantically relevant to the gold response. And then, it eliminates the exemplars that lexically resemble the gold responses to alleviate the dependency of the generative models on that exemplars. The remaining exemplars could be irrelevant to the given context since they are searched depending on the gold response. Thus, our proposed training method further utilizes the relevance scores between the given context and the exemplars to penalize the irrelevant exemplars. Extensive experiments demonstrate that our proposed training method alleviates the drawbacks of the existing exemplar-based generative models and significantly improves the performance in terms of appropriateness and informativeness.

cs.CL↗

arXiv2021

Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation

Despite the remarkable performance of large-scale generative models in open-domain conversation, they are known to be less practical for building real-time conversation systems due to high latency. On the other hand, retrieval models could return responses with much lower latency but show inferior performance to the large-scale generative models since the conversation quality is bounded by the pre-defined response set. To take advantage of both approaches, we propose a new training method called G2R (Generative-to-Retrieval distillation) that preserves the efficiency of a retrieval model while leveraging the conversational ability of a large-scale generative model by infusing the knowledge of the generative model into the retrieval model. G2R consists of two distinct techniques of distillation: the data-level G2R augments the dialogue dataset with additional responses generated by the large-scale generative model, and the model-level G2R transfers the response quality score assessed by the generative model to the score of the retrieval model by the knowledge distillation loss. Through extensive experiments including human evaluation, we demonstrate that our retrieval-based conversation system trained with G2R shows a substantially improved performance compared to the baseline retrieval model while showing significantly lower inference latency than the large-scale generative models.

cs.CL↗

arXiv2021

Disentangling Label Distribution for Long-tailed Visual Recognition

The current evaluation protocol of long-tailed visual recognition trains the classification model on the long-tailed source label distribution and evaluates its performance on the uniform target label distribution. Such protocol has questionable practicality since the target may also be long-tailed. Therefore, we formulate long-tailed visual recognition as a label shift problem where the target and source label distributions are different. One of the significant hurdles in dealing with the label shift problem is the entanglement between the source label distribution and the model prediction. In this paper, we focus on disentangling the source label distribution from the model prediction. We first introduce a simple but overlooked baseline method that matches the target label distribution by post-processing the model prediction trained by the cross-entropy loss and the Softmax function. Although this method surpasses state-of-the-art methods on benchmark datasets, it can be further improved by directly disentangling the source label distribution from the model prediction in the training phase. Thus, we propose a novel method, LAbel distribution DisEntangling (LADE) loss based on the optimal bound of Donsker-Varadhan representation. LADE achieves state-of-the-art performance on benchmark datasets such as CIFAR-100-LT, Places-LT, ImageNet-LT, and iNaturalist 2018. Moreover, LADE outperforms existing methods on various shifted target label distributions, showing the general adaptability of our proposed method.

cs.CV↗

arXiv2021

Self-Reorganizing and Rejuvenating CNNs for Increasing Model Capacity Utilization

In this paper, we propose self-reorganizing and rejuvenating convolutional neural networks; a biologically inspired method for improving the computational resource utilization of neural networks. The proposed method utilizes the channel activations of a convolution layer in order to reorganize that layers parameters. The reorganized parameters are clustered to avoid parameter redundancies. As such, redundant neurons with similar activations are merged leaving room for the remaining parameters to rejuvenate. The rejuvenated parameters learn different features to supplement those learned by the reorganized surviving parameters. As a result, the network capacity utilization increases improving the baseline network performance without any changes to the network structure. The proposed method can be applied to various network architectures during the training stage, or applied to a pre-trained model improving its performance. Experimental results showed that the proposed method is model-agnostic and can be applied to any backbone architecture increasing its performance due to the elevated utilization of the network capacity.

cs.LG↗

arXiv2020

Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding

On account of growing demands for personalization, the need for a so-called few-shot TTS system that clones speakers with only a few data is emerging. To address this issue, we propose Attentron, a few-shot TTS model that clones voices of speakers unseen during training. It introduces two special encoders, each serving different purposes. A fine-grained encoder extracts variable-length style information via an attention mechanism, and a coarse-grained encoder greatly stabilizes the speech synthesis, circumventing unintelligible gibberish even for synthesizing speech of unseen speakers. In addition, the model can scale out to an arbitrary number of reference audios to improve the quality of the synthesized speech. According to our experiments, including a human evaluation, the proposed model significantly outperforms state-of-the-art models when generating speech for unseen speakers in terms of speaker similarity and quality.

eess.AS↗

arXiv2017

Polarity-tunable magnetic tunnel junctions based on ferromagnetism at oxide heterointerfaces

Complex oxide systems have attracted considerable attention because of their fascinating properties, including the magnetic ordering at the conducting interface between two band insulators, such as LaAlO3 (LAO) and SrTiO3 (STO). However, the manipulation of the spin degree of freedom at the LAO/STO heterointerface has remained elusive. Here, we have fabricated hybrid magnetic tunnel junctions consisting of Co and LAO/STO ferromagnets with the insertion of a Ti layer in between, which clearly exhibit magnetic switching and the tunnelling magnetoresistance (TMR) effect below 10 K. The magnitude and the of the TMR are strongly dependent on the direction of the rotational magnetic field parallel to the LAO/STO plane, which is attributed to a strong Rashba-type spin orbit coupling in the LAO/STO heterostructure. Our study provides a further support for the existence of the macroscopic ferromagnetism at LAO/STO heterointerfaces and opens a novel route to realize interfacial spintronics devices.

cond-mat.str-el↗

arXiv2016

Spatiotemporal Evolution of Topological Order Upon Quantum Quench Across the Critical Point

We consider a topological superconducting wire and use the string order parameter to investigate the spatiotemporal evolution of the topological order upon a quantum quench across the critical point. We also analyze the propagation of the initially localized Majorana bound states after the quench, in order to examine the connection between the topological order and the unpaired Majorana states, which has been well established at equilibrium but remains illusive in dynamical situations. It is found that after the quench the string order parameters decay over a finite time and that the decaying behavior is universal, independent of the wire length and the final value of the chemical potential (the quenching parameter). It is also found that the topological order is revived repeatedly although the amplitude gradually decreases. Further, the topological order can propagate into the region which was initially in the non-topological state. It is observed that all these behaviors are in parallel and consistent with the propagation and dispersion of the Majorana wave functions. Finally, we propose a local probing method which can measure the non-local topological order.

cond-mat.mes-hall↗

arXiv2015

Emergent Universality in Nonequilibrium Processes of Critical Systems

We examine the Jarzynski equality for a quenching process across the critical point of second-order phase transitions, where absolute irreversibility and the effect of finite-sampling of the initial equilibrium distribution arise on an equal footing. We consider the Ising model as a prototypical example for spontaneous symmetry breaking and take into account the finite sampling issue by introducing a tolerance parameter. For a given tolerance parameter, the deviation from the Jarzynski equality depends onthe reduced coupling constant and the system size. In this work, we show that the deviation from the Jarzynski equality exhibits a universal scaling behavior inherited from the critical scaling laws of second-order phase transitions.

cond-mat.stat-mech↗

arXiv2015

The Kibble-Zurek Mechanism in a Topological Phase Transition

The Kibble-Zurek mechanism (KZM) is generalized to a class of multi-level systems and applied to study the quenching dynamics of one-dimensional (1D) topological superconductors (TS) with open ends. Unlike the periodic boundary condition, the open boundary condition, that is crucial for the zero-mode Majorana states localized at the boundaries, requires to consider many coupled levels. which is ultimately related to the zero-mode Majorana modes. Our generalized KZM predictions agree well with the numerically exact results for the 1D TS.

cond-mat.stat-mech↗