arXiv subjects

Doyoung Kim

Publications and source records attributed to Doyoung Kim.

25 records · Page 2Linked to original sources

arXiv2023

The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning

Language models (LMs) with less than 100B parameters are known to perform poorly on chain-of-thought (CoT) reasoning in contrast to large LMs when solving unseen tasks. In this work, we aim to equip smaller LMs with the step-by-step reasoning capability by instruction tuning with CoT rationales. In order to achieve this goal, we first introduce a new instruction-tuning dataset called the CoT Collection, which augments the existing Flan Collection (including only 9 CoT tasks) with additional 1.84 million rationales across 1,060 tasks. We show that CoT fine-tuning Flan-T5 (3B & 11B) with CoT Collection enables smaller LMs to have better CoT capabilities on unseen tasks. On the BIG-Bench-Hard (BBH) benchmark, we report an average improvement of +4.34% (Flan-T5 3B) and +2.60% (Flan-T5 11B), in terms of zero-shot task accuracy. Furthermore, we show that instruction tuning with CoT Collection allows LMs to possess stronger few-shot learning capabilities on 4 domain-specific tasks, resulting in an improvement of +2.24% (Flan-T5 3B) and +2.37% (Flan-T5 11B), even outperforming ChatGPT utilizing demonstrations until the max length by a +13.98% margin. Our code, the CoT Collection data, and model checkpoints are publicly available.

cs.CL↗

arXiv2023

Energy-Efficient Secure Offloading System Designed via UAV-Mounted Intelligent Reflecting Surface for Resilience Enhancement

With increasing interest in mmWave and THz communication systems, an unmanned aerial vehicle (UAV)-mounted intelligent reflecting surface (IRS) has been suggested as a key enabling technology to establish robust line-of-sight (LoS) connections with ground nodes owing to their free mobility and high altitude, especially for emergency and disaster response. This paper investigates a secure offloading system, where the UAV-mounted IRS assists the offloading procedures between ground users and an access point (AP) acting as an edge cloud. In this system, the users except the intended recipients in the offloading process are considered as potential eavesdroppers. The system aims to achieve the minimum total energy consumption of battery-limited ground user devices under constraints for secure offloading accomplishment and operability of UAV-mounted IRS, which is done by optimizing the transmit power of ground user devices, the trajectory and phase shift matrix of UAV-mounted IRS, and the offloading ratio between local execution and edge computing based on the successive convex approximation (SCA) algorithms. Numerical results show that the proposed algorithm can provide the considerable energy savings compared with local execution and partial optimizations.

eess.SP↗

arXiv2023

Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners

Meta-training, which fine-tunes the language model (LM) on various downstream tasks by maximizing the likelihood of the target label given the task instruction and input instance, has improved the zero-shot task generalization performance. However, meta-trained LMs still struggle to generalize to challenging tasks containing novel labels unseen during meta-training. In this paper, we propose Flipped Learning, an alternative method of meta-training which trains the LM to generate the task instruction given the input instance and label. During inference, the LM trained with Flipped Learning, referred to as Flipped, selects the label option that is most likely to generate the task instruction. On 14 tasks of the BIG-bench benchmark, the 11B-sized Flipped outperforms zero-shot T0-11B and even a 16 times larger 3-shot GPT-3 (175B) on average by 8.4% and 9.7% points, respectively. Flipped gives particularly large improvements on tasks with unseen labels, outperforming T0-11B by up to +20% average F1 score. This indicates that the strong task generalization of Flipped comes from improved generalization to novel labels. We release our code at https://github.com/seonghyeonye/Flipped-Learning.

cs.CL↗

arXiv2023

Exploring the Benefits of Training Expert Language Models over Instruction Tuning

Recently, Language Models (LMs) instruction-tuned on multiple tasks, also known as multitask-prompted fine-tuning (MT), have shown the capability to generalize to unseen tasks. Previous work has shown that scaling the number of training tasks is the key component in making stronger MT LMs. In this work, we report an unexpected finding that an expert LM fine-tuned on just a single task can outperform an MT LM trained with 300+ different tasks on 11 different unseen datasets and on 13 datasets of the BIG-bench benchmark by a mean accuracy of 3.20% and 1.29%, respectively. This finding casts doubt on the previously held belief that simply scaling the number of tasks makes stronger MT LMs. Leveraging this finding, we further show that this distributed approach of training a separate expert LM per training task instead of a single MT LM for zero-shot inference possesses many benefits including (1) avoiding negative task transfer that often occurs during instruction tuning, (2) being able to continually learn new tasks without having to re-train on previous tasks to avoid catastrophic forgetting, and (3) showing compositional capabilities when merging individual experts together. The code is available at https://github.com/joeljang/ELM.

cs.CL↗

arXiv2021

Highly-efficient broadband TE/TM polarization beam splitter

Highly-efficient broadband polarization beam splitters with small footprint and based on silicon photonics technology are of great importance in optical integrated circuits. In this paper, a crucially applicable optical component of a TE/TM polarization beam splitter has been designed and simulated using the inverse design of adjoint method. The mentioned device has ultra-high transmission efficiency (about 96% for TM$_{00}$ and 93% for TE$_{00}$ modes) and high extinction ratio of more than 16 dB over the bandwidth of 200 nm. Small footprint (2.2 $\times$ 2.1 $μ$m$^2$), broad bandwidth (200 nm), and compatibility with CMOS technology fabrication indicate the eligibility of this device.

physics.optics↗

arXiv2020

Band restructuring of ordered/disordered blue TiO2 for visible photocatalyst

Black TiO2 with/without noble metal has been proposed for visible photocatalyst, still leaving poor catalyst efficiency. Alternatively, phase-mixed TiO2 such as anatase and rutile has been commonly used for visible catalysts with the inevitable inclusion of noble metal. Here, we perform a noble metal-free visible photocatalyst blue TiO2 with type-II band-aligned ordered anatase/disordered rutile structure, via phase-selective reduction with alkali metals. The changed band alignment in this heterostructure was identified by absorption and ultraviolet photoemission spectroscopy, which was further confirmed by transient charge separation. The band alignment of type-I and type-II was clearly restructured by converting from ordered to disordered phase with a prolonged reduction period and as followed light absorbance enhancement also observed. Initiated type-I in a pristine sample, the type-II was organized from disordered rutile phase in 3-day Li-reduction. The type-II disordered rutile TiO2 heterostructure exhibits a remarkable photocatalytic performance by 55 times higher than conventional P25 TiO2 in solar-light driven hydrogen evolution reaction owing to an efficient electron and hole separation of type-II heterojunction. Furthermore, this restructured heterojunction type-II TiO2 demanded 10 times less Pt amount as a co-catalyst for the comparable photocatalytic performance, compared to Pt decorated type-I pristine anatase/rutile phase-mixed TiO2.

cond-mat.mtrl-sci↗

arXiv2019

Small Target Detection for Search and Rescue Operations using Distributed Deep Learning and Synthetic Data Generation

It is important to find the target as soon as possible for search and rescue operations. Surveillance camera systems and unmanned aerial vehicles (UAVs) are used to support search and rescue. Automatic object detection is important because a person cannot monitor multiple surveillance screens simultaneously for 24 hours. Also, the object is often too small to be recognized by the human eye on the surveillance screen. This study used UAVs around the Port of Houston and fixed surveillance cameras to build an automatic target detection system that supports the US Coast Guard (USCG) to help find targets (e.g., person overboard). We combined image segmentation, enhancement, and convolution neural networks to reduce detection time to detect small targets. We compared the performance between the auto-detection system and the human eye. Our system detected the target within 8 seconds, but the human eye detected the target within 25 seconds. Our systems also used synthetic data generation and data augmentation techniques to improve target detection accuracy. This solution may help the search and rescue operations of the first responders in a timely manner.

cs.CV↗