arXiv subjects

Hyunsoo Cho

Publications and source records attributed to Hyunsoo Cho.

35 records · Page 2Linked to original sources

arXiv2024

Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model

Many recent studies endeavor to improve open-source language models through imitation learning, and re-training on the synthetic instruction data from state-of-the-art proprietary models like ChatGPT and GPT-4. However, the innate nature of synthetic data inherently contains noisy data, giving rise to a substantial presence of low-quality data replete with erroneous responses, and flawed reasoning. Although we intuitively grasp the potential harm of noisy data, we lack a quantitative understanding of its impact. To this end, this paper explores the correlation between the degree of noise and its impact on language models through instruction tuning. We first introduce the Falsity-Controllable (FACO) dataset, which comprises pairs of true answers with corresponding reasoning, as well as false pairs to manually control the falsity ratio of the dataset.Through our extensive experiments, we found multiple intriguing findings of the correlation between the factuality of the dataset and instruction tuning: Specifically, we verified falsity of the instruction is highly relevant to various benchmark scores. Moreover, when LLMs are trained with false instructions, they learn to lie and generate fake unfaithful answers, even though they know the correct answer for the user request. Additionally, we noted that once the language model is trained with a dataset contaminated by noise, restoring its original performance is possible, but it failed to reach full performance.

cs.CL↗

arXiv2023

Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLP

When deploying machine learning systems to the wild, it is highly desirable for them to effectively leverage prior knowledge to the unfamiliar domain while also firing alarms to anomalous inputs. In order to address these requirements, Universal Domain Adaptation (UniDA) has emerged as a novel research area in computer vision, focusing on achieving both adaptation ability and robustness (i.e., the ability to detect out-of-distribution samples). While UniDA has led significant progress in computer vision, its application on language input still needs to be explored despite its feasibility. In this paper, we propose a comprehensive benchmark for natural language that offers thorough viewpoints of the model's generalizability and robustness. Our benchmark encompasses multiple datasets with varying difficulty levels and characteristics, including temporal shifts and diverse domains. On top of our testbed, we validate existing UniDA methods from computer vision and state-of-the-art domain adaptation techniques from NLP literature, yielding valuable findings: We observe that UniDA methods originally designed for image input can be effectively transferred to the natural language domain while also underscoring the effect of adaptation difficulty in determining the model's performance.

cs.CL↗

arXiv2023

Prompt-Augmented Linear Probing: Scaling beyond the Limit of Few-shot In-Context Learners

Through in-context learning (ICL), large-scale language models are effective few-shot learners without additional model fine-tuning. However, the ICL performance does not scale well with the number of available training samples as it is limited by the inherent input length constraint of the underlying language model. Meanwhile, many studies have revealed that language models are also powerful feature extractors, allowing them to be utilized in a black-box manner and enabling the linear probing paradigm, where lightweight discriminators are trained on top of the pre-extracted input representations. This paper proposes prompt-augmented linear probing (PALP), a hybrid of linear probing and ICL, which leverages the best of both worlds. PALP inherits the scalability of linear probing and the capability of enforcing language models to derive more meaningful representations via tailoring input into a more conceivable form. Throughout in-depth investigations on various datasets, we verified that PALP significantly enhances the input representations closing the gap between ICL in the data-hungry scenario and fine-tuning in the data-abundant scenario with little training overhead, potentially making PALP a strong alternative in a black-box scenario.

cs.CL↗

arXiv2023

Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning

As the size of the pre-trained language model (PLM) continues to increase, numerous parameter-efficient transfer learning methods have been proposed recently to compensate for the tremendous cost of fine-tuning. Despite the impressive results achieved by large pre-trained language models (PLMs) and various parameter-efficient transfer learning (PETL) methods on sundry benchmarks, it remains unclear if they can handle inputs that have been distributionally shifted effectively. In this study, we systematically explore how the ability to detect out-of-distribution (OOD) changes as the size of the PLM grows or the transfer methods are altered. Specifically, we evaluated various PETL techniques, including fine-tuning, Adapter, LoRA, and prefix-tuning, on three different intention classification tasks, each utilizing various language models with different scales.

cs.CL↗

arXiv2023

CELDA: Leveraging Black-box Language Model as Enhanced Classifier without Labels

Utilizing language models (LMs) without internal access is becoming an attractive paradigm in the field of NLP as many cutting-edge LMs are released through APIs and boast a massive scale. The de-facto method in this type of black-box scenario is known as prompting, which has shown progressive performance enhancements in situations where data labels are scarce or unavailable. Despite their efficacy, they still fall short in comparison to fully supervised counterparts and are generally brittle to slight modifications. In this paper, we propose Clustering-enhanced Linear Discriminative Analysis, a novel approach that improves the text classification accuracy with a very weak-supervision signal (i.e., name of the labels). Our framework draws a precise decision boundary without accessing weights or gradients of the LM model or data labels. The core ideas of CELDA are twofold: (1) extracting a refined pseudo-labeled dataset from an unlabeled dataset, and (2) training a lightweight and robust model on the top of LM, which learns an accurate decision boundary from an extracted noisy dataset. Throughout in-depth investigations on various datasets, we demonstrated that CELDA reaches new state-of-the-art in weakly-supervised text classification and narrows the gap with a fully-supervised model. Additionally, our proposed methodology can be applied universally to any LM and has the potential to scale to larger models, making it a more viable option for utilizing large LMs.

cs.CL↗

arXiv2023

Analyzing the Latent Space of GAN through Local Dimension Estimation

The impressive success of style-based GANs (StyleGANs) in high-fidelity image synthesis has motivated research to understand the semantic properties of their latent spaces. In this paper, we approach this problem through a geometric analysis of latent spaces as a manifold. In particular, we propose a local dimension estimation algorithm for arbitrary intermediate layers in a pre-trained GAN model. The estimated local dimension is interpreted as the number of possible semantic variations from this latent variable. Moreover, this intrinsic dimension estimation enables unsupervised evaluation of disentanglement for a latent space. Our proposed metric, called Distortion, measures an inconsistency of intrinsic tangent space on the learned latent space. Distortion is purely geometric and does not require any additional attribute information. Nevertheless, Distortion shows a high correlation with the global-basis-compatibility and supervised disentanglement score. Our work is the first step towards selecting the most disentangled latent space among various latent spaces in a GAN without attribute labels.

cs.CV↗

arXiv2023

Finding the global semantic representation in GAN through Frechet Mean

The ideally disentangled latent space in GAN involves the global representation of latent space with semantic attribute coordinates. In other words, considering that this disentangled latent space is a vector space, there exists the global semantic basis where each basis component describes one attribute of generated images. In this paper, we propose an unsupervised method for finding this global semantic basis in the intermediate latent space in GANs. This semantic basis represents sample-independent meaningful perturbations that change the same semantic attribute of an image on the entire latent space. The proposed global basis, called Fréchet basis, is derived by introducing Fréchet mean to the local semantic perturbations in a latent space. Fréchet basis is discovered in two stages. First, the global semantic subspace is discovered by the Fréchet mean in the Grassmannian manifold of the local semantic subspaces. Second, Fréchet basis is found by optimizing a basis of the semantic subspace via the Fréchet mean in the Special Orthogonal Group. Experimental results demonstrate that Fréchet basis provides better semantic factorization and robustness compared to the previous methods. Moreover, we suggest the basis refinement scheme for the previous methods. The quantitative experiments show that the refined basis achieves better semantic factorization while constrained on the same semantic subspace given by the previous method.

cs.CV↗

arXiv2023

Masked Contrastive Learning for Anomaly Detection

Detecting anomalies is one fundamental aspect of a safety-critical software system, however, it remains a long-standing problem. Numerous branches of works have been proposed to alleviate the complication and have demonstrated their efficiencies. In particular, self-supervised learning based methods are spurring interest due to their capability of learning diverse representations without additional labels. Among self-supervised learning tactics, contrastive learning is one specific framework validating their superiority in various fields, including anomaly detection. However, the primary objective of contrastive learning is to learn task-agnostic features without any labels, which is not entirely suited to discern anomalies. In this paper, we propose a task-specific variant of contrastive learning named masked contrastive learning, which is more befitted for anomaly detection. Moreover, we propose a new inference method dubbed self-ensemble inference that further boosts performance by leveraging the ability learned through auxiliary self-supervision tasks. By combining our models, we can outperform previous state-of-the-art methods by a significant margin on various benchmark datasets.

cs.LG↗

arXiv2022

Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations

Despite recent explosion of interests in in-context learning, the underlying mechanism and the precise impact of the quality of demonstrations remain elusive. Intuitively, ground-truth labels should have as much impact in in-context learning (ICL) as supervised learning, but recent work reported that the input-label correspondence is significantly less important than previously thought. Intrigued by this counter-intuitive observation, we re-examine the importance of ground-truth labels in in-context learning. With the introduction of two novel metrics, namely Label-Correctness Sensitivity and Ground-truth Label Effect Ratio (GLER), we were able to conduct quantifiable analysis on the impact of ground-truth label demonstrations. Through extensive analyses, we find that the correct input-label mappings can have varying impacts on the downstream in-context learning performances, depending on the experimental configuration. Through additional studies, we identify key components, such as the verbosity of prompt templates and the language model size, as the controlling factor to achieve more noise-resilient ICL.

cs.CL↗

arXiv2022

Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemble

Out-of-distribution (OOD) detection aims to discern outliers from the intended data distribution, which is crucial to maintaining high reliability and a good user experience. Most recent studies in OOD detection utilize the information from a single representation that resides in the penultimate layer to determine whether the input is anomalous or not. Although such a method is straightforward, the potential of diverse information in the intermediate layers is overlooked. In this paper, we propose a novel framework based on contrastive learning that encourages intermediate features to learn layer-specialized representations and assembles them implicitly into a single representation to absorb rich information in the pre-trained language model. Extensive experiments in various intent classification and OOD datasets demonstrate that our approach is significantly more effective than other works.

cs.CL↗

arXiv2022

Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator

Large-scale pre-trained language models (PLMs) are well-known for being capable of solving a task simply by conditioning a few input-label pairs dubbed demonstrations on a prompt without being explicitly tuned for the desired downstream task. Such a process (i.e., in-context learning), however, naturally leads to high reliance on the demonstrations which are usually selected from external datasets. In this paper, we propose self-generated in-context learning (SG-ICL), which generates demonstrations for in-context learning from PLM itself to minimize the reliance on the external demonstration. We conduct experiments on four different text classification tasks and show SG-ICL significantly outperforms zero-shot learning and is generally worth approximately 0.6 gold training samples. Moreover, our generated demonstrations show more consistent performance with low variance compared to randomly selected demonstrations from the training dataset.

cs.CL↗

arXiv2022

Combinatorics on bounded free Motzkin paths and its applications

In this paper, we construct a bijection from a set of bounded free Motzkin paths to a set of bounded Motzkin prefixes that induces a bijection from a set of bounded free Dyck paths to a set of bounded Dyck prefixes. We also give bijections between a set of bounded cornerless Motzkin paths and a set of $t$-core partitions, and a set of bounded cornerless symmetric Motzkin paths and a set of self-conjugate $t$-core partitions. As an application, we get explicit formulas for the number of ordinary and self-conjugate $t$-core partitions with a fixed number of corners.

math.CO↗

arXiv2022

Results on bar-core partitions, core shifted Young diagrams, and doubled distinct cores

Simultaneous bar-cores, core shifted Young diagrams (or CSYDs), and doubled distinct cores have been studied since Morris and Yaseen introduced the concept of bar-cores. In this paper, our goal is to give a formula for the number of these core partitions on $(s,t)$-cores and $(s,s+d,s+2d)$-cores for the remaining cases that are not covered yet. In order to achieve this goal, we observe a characterization of $\bar{s}$-core partitions to obtain characterizations of doubled distinct $s$-core partitions and $s$-CSYDs. By using them, we construct $NE$ lattice path interpretations of these core partitions on $(s,t)$-cores. Also, we give free Motzkin path interpretations of these core partitions on $(s,s+d,s+2d)$-cores.

math.CO↗

arXiv2020

Self-conjugate $(s,s+d,\dots,s+pd)$-core partitions and free rational Motzkin paths

A partition is called an $(s_1,s_2,\dots,s_p)$-core partition if it is simultaneously an $s_i$-core for all $i=1,2,\dots,p$. Simultaneous core partitions have been actively studied in various directions. In particular, researchers concerned with properties of such partitions when the sequence of $s_i$ is an arithmetic progression. In this paper, for $p\geq 2$ and relatively prime positive integers $s$ and $d$, we propose the $(s+d,d;a)$-abacus of a self-conjugate partition and establish a bijection between the set of self-conjugate $(s,s+d,\dots,s+pd)$-core partitions and the set of free rational Motzkin paths with appropriate conditions. For $p=2,3$, we give formulae for the number of self-conjugate $(s,s+d,\dots,s+pd)$-core partitions and the number of self-conjugate $(s,s+1,\dots,s+p)$-core partitions with $m$ corners.

math.CO↗

arXiv2020

The $(s,s+d,\dots,s+pd)$-core partitions and the rational Motzkin paths

In this paper, we propose an $(s+d,d)$-abacus for $(s,s+d,\dots,s+pd)$-core partitions and establish a bijection between the $(s,s+d,\dots,s+pd)$-core partitions and the rational Motzkin paths of type $(s+d,-d)$. This result not only gives a lattice path interpretation of the $(s,s+d,\dots,s+pd)$-core partitions but also counts them with a closed formula. Also we enumerate $(s,s+1,\dots,s+p)$-core partitions with $k$ corners and self-conjugate $(s,s+1,\dots,s+p)$-core partitions.

math.CO↗

arXiv2019

Counting self-conjugate (s,s+1,s+2)-core partitions

We are concerned with counting self-conjugate $(s,s+1,s+2)$-core partitions. A Motzkin path of length $n$ is a path from $(0,0)$ to $(n,0)$ which stays above the $x$-axis and consists of the up $U=(1,1)$, down $D=(1,-1)$, and flat $F=(1,0)$ steps. We say that a Motzkin path of length $n$ is symmetric if its reflection about the line $x=n/2$ is itself. In this paper, we show that the number of self-conjugate $(s,s+1,s+2)$-cores is equal to the number of symmetric Motzkin paths of length $s$, and give a closed formula for this number.

math.CO↗

arXiv2018

A bijection between self-conjugate and ordinary partitions and counting simultaneous cores as its application

We give a bijection between the set of self-conjugate partitions and that of ordinary partitions. Also, we show the relation between hook lengths of self conjugate partition and corresponding partition via the bijection. As a corollary, we give new combinatorial interpretations for the Catalan number and the Motzkin number in terms of self-conjugate simultaneous core partitions.

math.CO↗