arXiv subjects

Taeuk Kim

Publications and source records attributed to Taeuk Kim.

50 records · Page 3Linked to original sources

arXiv2021

Self-Guided Contrastive Learning for BERT Sentence Representations

Although BERT and its variants have reshaped the NLP landscape, it still remains unclear how best to derive sentence embeddings from such pre-trained Transformers. In this work, we propose a contrastive learning method that utilizes self-guidance for improving the quality of BERT sentence representations. Our method fine-tunes BERT in a self-supervised fashion, does not rely on data augmentation, and enables the usual [CLS] token embeddings to function as sentence vectors. Moreover, we redesign the contrastive learning objective (NT-Xent) and apply it to sentence representation learning. We demonstrate with extensive experiments that our approach is more effective than competitive baselines on diverse sentence-related tasks. We also show it is efficient at inference and robust to domain shifts.

cs.CL↗

arXiv2020

Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads

Transformer-based pre-trained language models (PLMs) have dramatically improved the state of the art in NLP across many tasks. This has led to substantial interest in analyzing the syntactic knowledge PLMs learn. Previous approaches to this question have been limited, mostly using test suites or probes. Here, we propose a novel fully unsupervised parsing approach that extracts constituency trees from PLM attention heads. We rank transformer attention heads based on their inherent properties, and create an ensemble of high-ranking heads to produce the final tree. Our method is adaptable to low-resource languages, as it does not rely on development sets, which can be expensive to annotate. Our experiments show that the proposed method often outperform existing approaches if there is no development set present. Our unsupervised parser can also be used as a tool to analyze the grammars PLMs learn implicitly. For this, we use the parse trees induced by our method to train a neural PCFG and compare it to a grammar derived from a human-annotated treebank.

cs.CL↗

arXiv2020

IDS at SemEval-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize?

We propose a novel method that enables us to determine words that deserve to be emphasized from written text in visual media, relying only on the information from the self-attention distributions of pre-trained language models (PLMs). With extensive experiments and analyses, we show that 1) our zero-shot approach is superior to a reasonable baseline that adopts TF-IDF and that 2) there exist several attention heads in PLMs specialized for emphasis selection, confirming that PLMs are capable of recognizing important words in sentences.

cs.CL↗

arXiv2020

Optimizing Placement of Heap Memory Objects in Energy-Constrained Hybrid Memory Systems

Main memory (DRAM) significantly impacts the power and energy utilization of the overall server system. Non-Volatile Memory (NVM) devices, such as Phase Change Memory and Spin-Transfer Torque RAM, are suitable candidates for main memory to reduce energy consumption. But unlike DRAM, NVMs access latencies are higher than DRAM and NVM writes are more energy sensitive than DRAM write operations. Thus, Hybrid Main Memory Systems (HMMS) employing DRAM and NVM have been proposed to reduce the overall energy depletion of main memory while optimizing the performance of NVM. This paper proposes eMap, an optimal heap memory object placement planner in HMMS. eMap considers the object-level access patterns and energy consumption at the application level and provides an ideal placement strategy for each object to augment performance and energy utilization. eMap is equipped with two modules, eMPlan and eMDyn. Specifically, eMPlan is a static placement planner which provides one time placement policies for memory object to meet the energy budget while eMDyn is a runtime placement planner to consider the change in energy limiting constraint during the runtime and shuffles the memory objects by taking into account the access patterns as well as the migration cost in terms of energy and performance. The evaluation shows that our proposed solution satisfies both the energy limiting constraint and the performance. We compare our methodology with the state-of-the-art memory object classification and allocation (MOCA) framework. Our extensive evaluation shows that our proposed solution, eMPlan meets the energy constraint with 4.17 times less costly and reducing the energy consumption up to 14% with the same performance. eMDyn also satisfies the performance and energy requirement while considering the migration cost in terms of time and energy.

cs.AR↗

arXiv2020

Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction

With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.

cs.CL↗

arXiv2019

Cell-aware Stacked LSTMs for Modeling Sentences

We propose a method of stacking multiple long short-term memory (LSTM) layers for modeling sentences. In contrast to the conventional stacked LSTMs where only hidden states are fed as input to the next layer, the suggested architecture accepts both hidden and memory cell states of the preceding layer and fuses information from the left and the lower context using the soft gating mechanism of LSTMs. Thus the architecture modulates the amount of information to be delivered not only in horizontal recurrence but also in vertical connections, from which useful features extracted from lower layers are effectively conveyed to upper layers. We dub this architecture Cell-aware Stacked LSTM (CAS-LSTM) and show from experiments that our models bring significant performance gain over the standard LSTMs on benchmark datasets for natural language inference, paraphrase detection, sentiment classification, and machine translation. We also conduct extensive qualitative analysis to understand the internal behavior of the suggested approach.

cs.CL↗

arXiv2019

Don't Just Scratch the Surface: Enhancing Word Representations for Korean with Hanja

We propose a simple yet effective approach for improving Korean word representations using additional linguistic annotation (i.e. Hanja). We employ cross-lingual transfer learning in training word representations by leveraging the fact that Hanja is closely related to Chinese. We evaluate the intrinsic quality of representations learned through our approach using the word analogy and similarity tests. In addition, we demonstrate their effectiveness on several downstream tasks, including a novel Korean news headline generation task.

cs.CL↗

arXiv2019

Summary Level Training of Sentence Rewriting for Abstractive Summarization

As an attempt to combine extractive and abstractive summarization, Sentence Rewriting models adopt the strategy of extracting salient sentences from a document first and then paraphrasing the selected ones to generate a summary. However, the existing models in this framework mostly rely on sentence-level rewards or suboptimal labels, causing a mismatch between a training objective and evaluation metric. In this paper, we present a novel training signal that directly maximizes summary-level ROUGE scores through reinforcement learning. In addition, we incorporate BERT into our model, making good use of its ability on natural language understanding. In extensive experiments, we show that a combination of our proposed model and training procedure obtains new state-of-the-art performance on both CNN/Daily Mail and New York Times datasets. We also demonstrate that it generalizes better on DUC-2002 test set.

cs.CL↗

arXiv2019

A Cross-Sentence Latent Variable Model for Semi-Supervised Text Sequence Matching

We present a latent variable model for predicting the relationship between a pair of text sequences. Unlike previous auto-encoding--based approaches that consider each sequence separately, our proposed framework utilizes both sequences within a single model by generating a sequence that has a given relationship with a source sequence. We further extend the cross-sentence generating framework to facilitate semi-supervised training. We also define novel semantic constraints that lead the decoder network to generate semantically plausible and diverse sequences. We demonstrate the effectiveness of the proposed model from quantitative and qualitative experiments, while achieving state-of-the-art results on semi-supervised natural language inference and paraphrase identification.

cs.CL↗

arXiv2018

Dynamic Compositionality in Recursive Neural Networks with Structure-aware Tag Representations

Most existing recursive neural network (RvNN) architectures utilize only the structure of parse trees, ignoring syntactic tags which are provided as by-products of parsing. We present a novel RvNN architecture that can provide dynamic compositionality by considering comprehensive syntactic information derived from both the structure and linguistic tags. Specifically, we introduce a structure-aware tag representation constructed by a separate tag-level tree-LSTM. With this, we can control the composition function of the existing word-level tree-LSTM by augmenting the representation as a supplementary input to the gate functions of the tree-LSTM. In extensive experiments, we show that models built upon the proposed architecture obtain superior or competitive performance on several sentence-level tasks such as sentiment analysis and natural language inference when compared against previous tree-structured models and other sophisticated neural models.

cs.CL↗

arXiv2018

SNU_IDS at SemEval-2018 Task 12: Sentence Encoder with Contextualized Vectors for Argument Reasoning Comprehension

We present a novel neural architecture for the Argument Reasoning Comprehension task of SemEval 2018. It is a simple neural network consisting of three parts, collectively judging whether the logic built on a set of given sentences (a claim, reason, and warrant) is plausible or not. The model utilizes contextualized word vectors pre-trained on large machine translation (MT) datasets as a form of transfer learning, which can help to mitigate the lack of training data. Quantitative analysis shows that simply leveraging LSTMs trained on MT datasets outperforms several baselines and non-transferred models, achieving accuracies of about 70% on the development set and about 60% on the test set.

cs.CL↗

arXiv2018

FT-LADS: Fault-Tolerant Object-Logging based Big Data Transfer System using Layout-Aware Data Scheduling

Layout-Aware Data Scheduler (LADS) data transfer tool, identifies and addresses the issues that lead to congestion on the path of an end-to-end data transfer in the terabit network environments. It exploits the underlying storage layout at each endpoint to maximize throughput without negatively impacting the performance of shared storage resources for other users. LADS can avoid congested storage elements within the shared storage resource, improving input/output bandwidth, and hence the data transfer rates across the high speed networks. However, absence of FT (fault tolerance) support in LADS results in data re-transmission overhead along with the possible integrity issues upon errors. In this paper, we propose object based logging methods to avoid transmitting the objects which are successfully written to Parallel File System (PFS) at the sink end. Depending on the number of logger files created, for the whole dataset, we classified our fault tolerance mechanisms into three different categories: File logger, Transaction logger and Universal logger. Also, to address space overhead of these object based logging mechanisms, we have proposed different methods of populating logger files with the information of the completed objects. We have evaluated the data transfer performance and recovery time overhead of the proposed object based logging fault tolerant mechanisms on LADS data transfer tool. Our experimental results show that, LADS in conjunction with proposed object based fault tolerance mechanisms exhibit an overhead of less than 1% with respect to data transfer time and total recovery time overhead is around 10% of total data transfer time at any fault point.

cs.DC↗

arXiv2018

SCISPACE: A Scientific Collaboration Workspace for File Systems in Geo-Distributed HPC Data Centers

Future terabit networks are committed to dramatically improving big data motion between geographically dispersed HPC data centers.The scientific community takes advantage of the terabit networks such as DOE's ESnet and accelerates the trend to build a small world of collaboration between geospatial HPC data centers. It improves information and resource sharing for joint simulation and analysis between the HPC data centers. In this paper, we propose to build SCISPACE (Scientific Collaboration Workspace) for collaborative data centers. It provides a global view of information shared from multiple geo-distributed HPC data centers under a single workspace. SCISPACE supports native data-access to gain high-performance when data read or write is required in native data center namespace. It is accomplished by integrating a metadata export protocol. To optimize scientific collaborations across HPC data centers, SCISPACE implements search and discovery service. To evaluate, we configured two geo-distributed small-scale HPC data centers connected via high-speed Infiniband network, equipped with LustreFS. We show the feasibility of SCISPACE using real scientific datasets and applications. The evaluation results show average 36\% performance boost when the proposed native-data access is employed in collaborations.

cs.DC↗

arXiv2017

A Syllable-based Technique for Word Embeddings of Korean Words

Word embedding has become a fundamental component to many NLP tasks such as named entity recognition and machine translation. However, popular models that learn such embeddings are unaware of the morphology of words, so it is not directly applicable to highly agglutinative languages such as Korean. We propose a syllable-based learning model for Korean using a convolutional neural network, in which word representation is composed of trained syllable vectors. Our model successfully produces morphologically meaningful representation of Korean words compared to the original Skip-gram embeddings. The results also show that it is quite robust to the Out-of-Vocabulary problem.

cs.CL↗