arXiv subjects

Jaewook Lee

Publications and source records attributed to Jaewook Lee.

69 records · Page 4Linked to original sources

arXiv2023

A Conceptual Model for End-to-End Causal Discovery in Knowledge Tracing

In this paper, we take a preliminary step towards solving the problem of causal discovery in knowledge tracing, i.e., finding the underlying causal relationship among different skills from real-world student response data. This problem is important since it can potentially help us understand the causal relationship between different skills without extensive A/B testing, which can potentially help educators to design better curricula according to skill prerequisite information. Specifically, we propose a conceptual solution, a novel causal gated recurrent unit (GRU) module in a modified deep knowledge tracing model, which uses i) a learnable permutation matrix for causal ordering among skills and ii) an optionally learnable lower-triangular matrix for causal structure among skills. We also detail how to learn the model parameters in an end-to-end, differentiable way. Our solution placed among the top entries in Task 3 of the NeurIPS 2022 Challenge on Causal Insights for Learning Paths in Education. We detail preliminary experiments as evaluated on the challenge's public leaderboard since the ground truth causal structure has not been publicly released, making detailed local evaluation impossible.

cs.LG↗

arXiv2023

Knowledge Graph-Augmented Korean Generative Commonsense Reasoning

Generative commonsense reasoning refers to the task of generating acceptable and logical assumptions about everyday situations based on commonsense understanding. By utilizing an existing dataset such as Korean CommonGen, language generation models can learn commonsense reasoning specific to the Korean language. However, language models often fail to consider the relationships between concepts and the deep knowledge inherent to concepts. To address these limitations, we propose a method to utilize the Korean knowledge graph data for text generation. Our experimental result shows that the proposed method can enhance the efficiency of Korean commonsense inference, thereby underlining the significance of employing supplementary data.

cs.CL↗

arXiv2023

Tighter Lower Bounds for Shuffling SGD: Random Permutations and Beyond

We study convergence lower bounds of without-replacement stochastic gradient descent (SGD) for solving smooth (strongly-)convex finite-sum minimization problems. Unlike most existing results focusing on final iterate lower bounds in terms of the number of components $n$ and the number of epochs $K$, we seek bounds for arbitrary weighted average iterates that are tight in all factors including the condition number $κ$. For SGD with Random Reshuffling, we present lower bounds that have tighter $κ$ dependencies than existing bounds. Our results are the first to perfectly close the gap between lower and upper bounds for weighted average iterates in both strongly-convex and convex cases. We also prove weighted average iterate lower bounds for arbitrary permutation-based SGD, which apply to all variants that carefully choose the best permutation. Our bounds improve the existing bounds in factors of $n$ and $κ$ and thereby match the upper bounds shown for a recently proposed algorithm called GraB.

cs.LG↗

arXiv2023

Differentially Private Sharpness-Aware Training

Training deep learning models with differential privacy (DP) results in a degradation of performance. The training dynamics of models with DP show a significant difference from standard training, whereas understanding the geometric properties of private learning remains largely unexplored. In this paper, we investigate sharpness, a key factor in achieving better generalization, in private learning. We show that flat minima can help reduce the negative effects of per-example gradient clipping and the addition of Gaussian noise. We then verify the effectiveness of Sharpness-Aware Minimization (SAM) for seeking flat minima in private learning. However, we also discover that SAM is detrimental to the privacy budget and computational time due to its two-step optimization. Thus, we propose a new sharpness-aware training method that mitigates the privacy-optimization trade-off. Our experimental results demonstrate that the proposed method improves the performance of deep learning models with DP from both scratch and fine-tuning. Code is available at https://github.com/jinseongP/DPSAT.

cs.LG↗

arXiv2023

SmartPhone: Exploring Keyword Mnemonic with Auto-generated Verbal and Visual Cues

In second language vocabulary learning, existing works have primarily focused on either the learning interface or scheduling personalized retrieval practices to maximize memory retention. However, the learning content, i.e., the information presented on flashcards, has mostly remained constant. Keyword mnemonic is a notable learning strategy that relates new vocabulary to existing knowledge by building an acoustic and imagery link using a keyword that sounds alike. Beyond that, producing verbal and visual cues associated with the keyword to facilitate building these links requires a manual process and is not scalable. In this paper, we explore an opportunity to use large language models to automatically generate verbal and visual cues for keyword mnemonics. Our approach, an end-to-end pipeline for auto-generating verbal and visual cues, can automatically generate highly memorable cues. We investigate the effectiveness of our approach via a human participant experiment by comparing it with manually generated cues.

cs.CL↗

arXiv2023

Unraveling the MEV Enigma: ABI-Free Detection Model using Graph Neural Networks

The detection of Maximal Extractable Value (MEV) in blockchain is crucial for enhancing blockchain security, as it enables the evaluation of potential consensus layer risks, the effectiveness of anti-centralization solutions, and the assessment of user exploitation. However, existing MEV detection methods face limitations due to their low recall rate, reliance on pre-registered Application Binary Interfaces (ABIs) and the need for continuous monitoring of new DeFi services. In this paper, we propose ArbiNet, a novel GNN-based detection model that offers a low-overhead and accurate solution for MEV detection without requiring knowledge of smart contract code or ABIs. We collected an extensive MEV dataset, surpassing currently available public datasets, to train ArbiNet. Our implemented model and open dataset enhance the understanding of the MEV landscape, serving as a foundation for MEV quantification and improved blockchain security.

cs.CR↗

arXiv2023

Neural Networks vs. Splines: Advances in Numerical Extruder Design

We present a novel application of neural networks to design improved mixing elements for single-screw extruders. Specifically, we propose to use neural networks in numerical shape optimization to parameterize geometries. Geometry parameterization is crucial in enabling efficient shape optimization as it allows for optimizing complex shapes using only a few design variables. Recent approaches often utilize CAD data in conjunction with spline-based methods where the spline's control points serve as design variables. Consequently, these approaches rely on the same design variables as specified by the human designer. While this choice is convenient, it either restricts the design to small modifications of given, initial design features - effectively prohibiting topological changes - or yields undesirably many design variables. In this work, we step away from CAD and spline-based approaches and construct an artificial, feature-dense yet low-dimensional optimization space using a generative neural network. Using the neural network for the geometry parameterization extends state-of-the-art methods in that the resulting design space is not restricted to user-prescribed modifications of certain basis shapes. Instead, within the same optimization space, we can interpolate between and explore seemingly unrelated designs. To show the performance of this new approach, we integrate the developed shape parameterization into our numerical design framework for dynamic mixing elements in plastics extrusion. Finally, we challenge the novel method in a competitive setting against current free-form deformation-based approaches and demonstrate the method's performance even at this early stage.

cs.CE↗

arXiv2023

Exploring the Effect of Multi-step Ascent in Sharpness-Aware Minimization

Recently, Sharpness-Aware Minimization (SAM) has shown state-of-the-art performance by seeking flat minima. To minimize the maximum loss within a neighborhood in the parameter space, SAM uses an ascent step, which perturbs the weights along the direction of gradient ascent with a given radius. While single-step or multi-step can be taken during ascent steps, previous studies have shown that multi-step ascent SAM rarely improves generalization performance. However, this phenomenon is particularly interesting because the multi-step ascent is expected to provide a better approximation of the maximum neighborhood loss. Therefore, in this paper, we analyze the effect of the number of ascent steps and investigate the difference between both single-step ascent SAM and multi-step ascent SAM. We identify the effect of the number of ascent on SAM optimization and reveal that single-step ascent SAM and multi-step ascent SAM exhibit distinct loss landscapes. Based on these observations, we finally suggest a simple modification that can mitigate the inefficiency of multi-step ascent SAM.

cs.LG↗

arXiv2022

Towards Semi-automatic Detection and Localization of Indoor Accessibility Issues using Mobile Depth Scanning and Computer Vision

To help improve the safety and accessibility of indoor spaces, researchers and health professionals have created assessment instruments that enable homeowners and trained experts to audit and improve homes. With advances in computer vision, augmented reality (AR), and mobile sensors, new approaches are now possible. We introduce RASSAR (Room Accessibility and Safety Scanning in Augmented Reality), a new proof-of-concept prototype for semi-automatically identifying, categorizing, and localizing indoor accessibility and safety issues using LiDAR + camera data, machine learning, and AR. We present an overview of the current RASSAR prototype and a preliminary evaluation in a single home.

cs.HC↗

arXiv2022

Comment on Transferability and Input Transformation with Additive Noise

Adversarial attacks have verified the existence of the vulnerability of neural networks. By adding small perturbations to a benign example, adversarial attacks successfully generate adversarial examples that lead misclassification of deep learning models. More importantly, an adversarial example generated from a specific model can also deceive other models without modification. We call this phenomenon ``transferability". Here, we analyze the relationship between transferability and input transformation with additive noise by mathematically proving that the modified optimization can produce more transferable adversarial examples.

cs.LG↗

arXiv2022

First-principles theory of extending the spin qubit coherence time in hexagonal boron nitride

Negatively charged boron vacancies (VB-) in hexagonal boron nitride (h-BN) are a rapidly developing qubit platform in two-dimensional materials for solid-state quantum applications. However, their spin coherence time (T2) is very short, limited to a few microseconds owing to the inherently dense nuclear spin bath of the h-BN host. As the coherence time is one of the most fundamental properties of spin qubits, the short T2 time of VB- could significantly limit its potential as a promising spin qubit candidate. In this study, we theoretically proposed two materials engineering methods, which can substantially extend the T2 time of the VB- spin by four times more than its intrinsic T2. We performed quantum many-body computations by combining density functional theory and cluster correlation expansion and showed that replacing all the boron atoms in h-BN with the 10B isotope leads to the coherence enhancement of the VB- spin by a factor of three. In addition, the T2 time of the VB- can be enhanced by a factor of 1.3 by inducing a curvature around VB-. Herein, we elucidate that the curvature-induced inhomogeneous strain creates spatially varying quadrupole nuclear interactions, which effectively suppress the nuclear spin flip-flop dynamics in the bath. Importantly, we find that the combination of isotopic enrichment and strain engineering can maximize the VB- T2, yielding 207.2 and 161.9 μs for single- and multi-layer h-10BN, respectively. Furthermore, our results can be applied to any spin qubit in h-BN, strengthening their potential as material platforms to realize high-precision quantum sensors, quantum spin registers, and atomically thin quantum magnets.

cond-mat.mes-hall↗

arXiv2021

Bridged Adversarial Training

Adversarial robustness is considered as a required property of deep neural networks. In this study, we discover that adversarially trained models might have significantly different characteristics in terms of margin and smoothness, even they show similar robustness. Inspired by the observation, we investigate the effect of different regularizers and discover the negative effect of the smoothness regularizer on maximizing the margin. Based on the analyses, we propose a new method called bridged adversarial training that mitigates the negative effect by bridging the gap between clean and adversarial examples. We provide theoretical and empirical evidence that the proposed method provides stable and better robustness, especially for large perturbations.

cs.LG↗

arXiv2021

GradDiv: Adversarial Robustness of Randomized Neural Networks via Gradient Diversity Regularization

Deep learning is vulnerable to adversarial examples. Many defenses based on randomized neural networks have been proposed to solve the problem, but fail to achieve robustness against attacks using proxy gradients such as the Expectation over Transformation (EOT) attack. We investigate the effect of the adversarial attacks using proxy gradients on randomized neural networks and demonstrate that it highly relies on the directional distribution of the loss gradients of the randomized neural network. We show in particular that proxy gradients are less effective when the gradients are more scattered. To this end, we propose Gradient Diversity (GradDiv) regularizations that minimize the concentration of the gradients to build a robust randomized neural network. Our experiments on MNIST, CIFAR10, and STL10 show that our proposed GradDiv regularizations improve the adversarial robustness of randomized neural networks against a variety of state-of-the-art attack methods. Moreover, our method efficiently reduces the transferability among sample models of randomized neural networks.

cs.LG↗

arXiv2020

Understanding Catastrophic Overfitting in Single-step Adversarial Training

Although fast adversarial training has demonstrated both robustness and efficiency, the problem of "catastrophic overfitting" has been observed. This is a phenomenon in which, during single-step adversarial training, the robust accuracy against projected gradient descent (PGD) suddenly decreases to 0% after a few epochs, whereas the robust accuracy against fast gradient sign method (FGSM) increases to 100%. In this paper, we demonstrate that catastrophic overfitting is very closely related to the characteristic of single-step adversarial training which uses only adversarial examples with the maximum perturbation, and not all adversarial examples in the adversarial direction, which leads to decision boundary distortion and a highly curved loss surface. Based on this observation, we propose a simple method that not only prevents catastrophic overfitting, but also overrides the belief that it is difficult to prevent multi-step adversarial attacks with single-step adversarial training.

cs.LG↗

arXiv2013

Embedding Noise Prediction into List-Viterbi Decoding using Error Detection Codes for Magnetic Tape Systems

A List Viterbi detector produces a rank ordered list of the N globally best candidates in a trellis search. A List Viterbi detector structure is proposed that incorporates the noise prediction with periodic state-metric updates based on outer error detection codes (EDCs). More specifically, a periodic decision making process is utilized for a non-overlapping sliding windows of P bits based on the use of outer EDCs. In a number of magnetic recording applications, Error Correction Coding (ECC) is adversely effected by the presence of long and dominant error events. Unlike the conventional post processing methods that are usually tailored to a specific set of dominant error events or the joint modulation code trellis architectures that are operating on larger state spaces at the expense of increased implementation complexity, the proposed detector does not use any a priori information about the error event distributions and operates at reduced state trellis. We present pre ECC bit error rate performance as well as the post ECC codeword failure rates of the proposed detector using perfect detection scenario as well as practical detection codes as the EDCs are not essential to the overall design. Furthermore, it is observed that proposed algorithm does not introduce new error events. Simulation results show that the proposed algorithm gives improved bit error and post ECC codeword failure rates at the expense of some increase in complexity.

cs.IT↗