arXiv subjects

Jaewon Lee

Publications and source records attributed to Jaewon Lee.

At least 55 records · Page 3Linked to original sources

arXiv2024

The Llama 3 Herd of Models

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

cs.AI↗

arXiv2024

Self-supervised denoising of visual field data improves detection of glaucoma progression

Perimetric measurements provide insight into a patient's peripheral vision and day-to-day functioning and are the main outcome measure for identifying progression of visual damage from glaucoma. However, visual field data can be noisy, exhibiting high variance, especially with increasing damage. In this study, we demonstrate the utility of self-supervised deep learning in denoising visual field data from over 4000 patients to enhance its signal-to-noise ratio and its ability to detect true glaucoma progression. We deployed both a variational autoencoder (VAE) and a masked autoencoder to determine which self-supervised model best smooths the visual field data while reconstructing salient features that are less noisy and more predictive of worsening disease. Our results indicate that including a categorical p-value at every visual field location improves the smoothing of visual field data. Masked autoencoders led to cleaner denoised data than previous methods, such as variational autoencoders. A 4.7% increase in detection of progressing eyes with pointwise linear regression (PLR) was observed. The masked and variational autoencoders' smoothed data predicted glaucoma progression 2.3 months earlier when p-values were included compared to when they were not. The faster prediction of time to progression (TTP) and the higher percentage progression detected support our hypothesis that masking out visual field elements during training while including p-values at each location would improve the task of detection of visual field progression. Our study has clinically relevant implications regarding masking when training neural networks to denoise visual field data, resulting in earlier and more accurate detection of glaucoma progression. This denoising model can be integrated into future models for visual field analysis to enhance detection of glaucoma progression.

eess.IV↗

arXiv2024

DGS-SLAM: Gaussian Splatting SLAM in Dynamic Environment

We introduce Dynamic Gaussian Splatting SLAM (DGS-SLAM), the first dynamic SLAM framework built on the foundation of Gaussian Splatting. While recent advancements in dense SLAM have leveraged Gaussian Splatting to enhance scene representation, most approaches assume a static environment, making them vulnerable to photometric and geometric inconsistencies caused by dynamic objects. To address these challenges, we integrate Gaussian Splatting SLAM with a robust filtering process to handle dynamic objects throughout the entire pipeline, including Gaussian insertion and keyframe selection. Within this framework, to further improve the accuracy of dynamic object removal, we introduce a robust mask generation method that enforces photometric consistency across keyframes, reducing noise from inaccurate segmentation and artifacts such as shadows. Additionally, we propose the loop-aware window selection mechanism, which utilizes unique keyframe IDs of 3D Gaussians to detect loops between the current and past frames, facilitating joint optimization of the current camera poses and the Gaussian map. DGS-SLAM achieves state-of-the-art performance in both camera tracking and novel view synthesis on various dynamic SLAM benchmarks, proving its effectiveness in handling real-world dynamic scenes.

cs.RO↗

arXiv2024

Losses resistant verification of quantum non-Gaussian photon statistics

Quantum non-Gaussian states of light have fundamental properties that are essential for a multitude of applications in quantum technology. However, many of these features are difficult to detect using standard criteria due to optical losses and detector inefficiency. As the statistics of light are unknown, the loss correction on the data is unreliable, despite the fact that the losses can be precisely measured. To address this issue, we employ a loss-mitigated verification technique utilising quantum non-Gaussian witnesses, which incorporate the known optical losses and detector inefficiency into their derivation. This approach allows us to address the considerable challenge of experimentally demonstrating unheralded quantum non-Gaussian states of single photons and photon pairs.

quant-ph↗

arXiv2024

Obstructing two-torsion in the rational knot concordance group

It is well known that there are many 2-torsion elements in the classical knot concordance group. On the other hand, it is not known if there is any torsion element in the rational knot concordance group $\mathcal{C}_\mathbb{Q}$. Cha defined the algebraic rational concordance group $\mathcal{AC}_\mathbb{Q}$, an analogue of the classical algebraic concordance group, and showed that $\mathcal{AC}_\mathbb{Q}\cong\mathbb{Z}^\infty\oplus\mathbb{Z}_2^\infty\oplus\mathbb{Z}_4^\infty$. The knots that represent 2-torsions in $\mathcal{AC}_\mathbb{Q}$ potentially have order $2$ in $\mathcal{C}_\mathbb{Q}$. In this paper, we provide an obstruction for knots of order $2$ in $\mathcal{AC}_\mathbb{Q}$ from being of finite order in $\mathcal{C}_\mathbb{Q}$. Moreover, we give a family consisting of such knots that generates an infinite rank subgroup of $\mathcal{C}_\mathbb{Q}$. We also note that Cha proved that in higher dimensions, the algebraic rational concordance order is the same as the rational knot concordance order. Our obstruction is based on the localized von Neumann $ρ$-invariant.

math.GT↗

arXiv2024

SUTRA: Scalable Multilingual Language Model Architecture

In this paper, we introduce SUTRA, multilingual Large Language Model architecture capable of understanding, reasoning, and generating text in over 50 languages. SUTRA's design uniquely decouples core conceptual understanding from language-specific processing, which facilitates scalable and efficient multilingual alignment and learning. Employing a Mixture of Experts framework both in language and concept processing, SUTRA demonstrates both computational efficiency and responsiveness. Through extensive evaluations, SUTRA is demonstrated to surpass existing models like GPT-3.5, Llama2 by 20-30% on leading Massive Multitask Language Understanding (MMLU) benchmarks for multilingual tasks. SUTRA models are also online LLMs that can use knowledge from the internet to provide hallucination-free, factual and up-to-date responses while retaining their multilingual capabilities. Furthermore, we explore the broader implications of its architecture for the future of multilingual AI, highlighting its potential to democratize access to AI technology globally and to improve the equity and utility of AI in regions with predominantly non-English languages. Our findings suggest that SUTRA not only fills pivotal gaps in multilingual model capabilities but also establishes a new benchmark for operational efficiency and scalability in AI applications.

cs.CL↗

arXiv2024

Achieving Optical Refractive Index of 10-Plus by Colloidal Self-Assembly

This study demonstrates the developments of self-assembled optical metasurfaces to overcome inherent limitations in polarization density (P) within natural materials, which hinder achieving high refractive indices (n) at optical frequencies. The Maxwellian macroscopic description establishes a link between P and n, revealing a static limit in natural materials, restricting n to approximately 4.0 at optical frequencies. Optical metasurfaces, utilizing metallic colloids on a deep-subwavelength scale, offer a solution by unnaturally enhancing n through electric dipolar (ED) resonances. Self-assembly enables the creation of nanometer-scale metallic gaps between metallic nanoparticles (NPs), paving the way for achieving exceptionally high n at optical frequencies. This study focuses on assembling polyhedral gold (Au) NPs into a closely packed monolayer by rationally designing the polymeric ligand to balance attractive and repulsive forces, in that polymeric brush-mediated self-assembly of the close-packed Au NP monolayer is robustly achieved over a large-area. The resulting monolayer of Au nanospheres (NSs), nanooctahedras (NOs), and nanocubes (NCs) exhibits high macroscopic integrity and crystallinity, sufficiently enough for pushing n to record-high regimes. The study underlies the significance of capacitive coupling in achieving an unnaturally high n and explores fine-tuning Au NC size to optimize this coupling. The achieved n of 10.12 at optical frequencies stands as a benchmark, highlighting the potential of polyhedral Au NPs in advancing optical metasurfaces.

physics.optics↗

arXiv2024

Impact of temporal correlations, coherence, and postselection on two-photon interference

Two-photon interference is an indispensable resource in quantum photonics, but it is not straightforward to achieve. The cascaded generation of photon pairs contains intrinsic temporal correlations that negatively affect the ability of such sources to perform two-photon interference, thus hindering applications. We report on how such correlation interplays with decoherence and temporal postselection, and under which conditions temporal postselection could improve two-photon interference visibility. Our study identifies crucial parameters and points the way to a source with optimal performance.

quant-ph↗

arXiv2024

Implicit Neural Image Stitching

Existing frameworks for image stitching often provide visually reasonable stitchings. However, they suffer from blurry artifacts and disparities in illumination, depth level, etc. Although the recent learning-based stitchings relax such disparities, the required methods impose sacrifice of image qualities failing to capture high-frequency details for stitched images. To address the problem, we propose a novel approach, implicit Neural Image Stitching (NIS) that extends arbitrary-scale super-resolution. Our method estimates Fourier coefficients of images for quality-enhancing warps. Then, the suggested model blends color mismatches and misalignment in the latent space and decodes the features into RGB values of stitched images. Our experiments show that our approach achieves improvement in resolving the low-definition imaging of the previous deep image stitching with favorable accelerated image-enhancing methods. Our source code is available at https://github.com/minshu-kim/NIS.

cs.CV↗

arXiv2023

Semantic-Aware Implicit Template Learning via Part Deformation Consistency

Learning implicit templates as neural fields has recently shown impressive performance in unsupervised shape correspondence. Despite the success, we observe current approaches, which solely rely on geometric information, often learn suboptimal deformation across generic object shapes, which have high structural variability. In this paper, we highlight the importance of part deformation consistency and propose a semantic-aware implicit template learning framework to enable semantically plausible deformation. By leveraging semantic prior from a self-supervised feature extractor, we suggest local conditioning with novel semantic-aware deformation code and deformation consistency regularizations regarding part deformation, global deformation, and global scaling. Our extensive experiments demonstrate the superiority of the proposed method over baselines in various tasks: keypoint transfer, part label transfer, and texture transfer. More interestingly, our framework shows a larger performance gain under more challenging settings. We also provide qualitative analyses to validate the effectiveness of semantic-aware deformation. The code is available at https://github.com/mlvlab/PDC.

cs.CV↗

arXiv2023

RV-CURE: A RISC-V Capability Architecture for Full Memory Safety

Despite decades of efforts to resolve, memory safety violations are still persistent and problematic in modern systems. Various defense mechanisms have been proposed, but their deployment in real systems remains challenging because of performance, security, or compatibility concerns. In this paper, we propose RV-CURE, a RISC-V capability architecture that implements full-system support for full memory safety. For capability enforcement, we first propose a compiler technique, data-pointer tagging (DPT), applicable to protecting all memory types. It inserts a pointer tag in a pointer address and associates that tag with the pointer's capability metadata. DPT enforces a capability check for every memory access by a tagged pointer and thereby prevents illegitimate memory accesses. Furthermore, we investigate and present lightweight hardware extensions for DPT based on the open-source RISC-V BOOM processor. We observe that a capability-execution pipeline can be implemented in parallel with the existing memory-execution pipeline without intrusive modifications. With our seamless hardware integration, we achieve low-cost capability checks transparently performed in hardware. Altogether, we prototype RV-CURE as a synthesized RTL processor and conduct full-system evaluations on FPGAs running Linux OS. Our evaluations show that RV-CURE achieves strong memory safety at a 10.8% slowdown across the SPEC 2017 C/C++ workloads.

cs.AR↗

arXiv2023

Chemical Property-Guided Neural Networks for Naphtha Composition Prediction

The naphtha cracking process heavily relies on the composition of naphtha, which is a complex blend of different hydrocarbons. Predicting the naphtha composition accurately is crucial for efficiently controlling the cracking process and achieving maximum performance. Traditional methods, such as gas chromatography and true boiling curve, are not feasible due to the need for pilot-plant-scale experiments or cost constraints. In this paper, we propose a neural network framework that utilizes chemical property information to improve the performance of naphtha composition prediction. Our proposed framework comprises two parts: a Watson K factor estimation network and a naphtha composition prediction network. Both networks share a feature extraction network based on Convolutional Neural Network (CNN) architecture, while the output layers use Multi-Layer Perceptron (MLP) based networks to generate two different outputs - Watson K factor and naphtha composition. The naphtha composition is expressed in percentages, and its sum should be 100%. To enhance the naphtha composition prediction, we utilize a distillation simulator to obtain the distillation curve from the naphtha composition, which is dependent on its chemical properties. By designing a loss function between the estimated and simulated Watson K factors, we improve the performance of both Watson K estimation and naphtha composition prediction. The experimental results show that our proposed framework can predict the naphtha composition accurately while reflecting real naphtha chemical properties.

cs.LG↗

arXiv2023

Correlation study of temporal and emission properties of quiescent magnetars

We measured temporal and emission properties of quiescent magnetars using archival Chandra and XMM-Newton data, produced a list of the properties for 17 magnetars, and revisited previously suggested correlations between the properties. Our studies carried out with a larger sample, better spectral characterizations, and more thorough analyses not only confirmed previously-suggested correlations but also found new ones. The observed correlations differ from those seen in other neutron-star populations but generally accord with magnetar models. Specifically, the trends of the intriguing correlations of blackbody luminosity ($L_{\rm BB}$) with the spin-inferred dipole magnetic field strength ($B_{\rm S}$) and characteristic age ($τ_{\rm c}$) were measured to be $L_{\rm BB}\propto B_{\rm S}^{1.5}$ and $L_{\rm BB}\propto τ_{\rm c}^{-0.6}$, supporting the twisted magnetosphere and magnetothermal evolution models for magnetars. We report the analysis results and discuss our findings in the context of magnetar models.

astro-ph.HE↗

arXiv2023

Semantic-aware Occlusion Filtering Neural Radiance Fields in the Wild

We present a learning framework for reconstructing neural scene representations from a small number of unconstrained tourist photos. Since each image contains transient occluders, decomposing the static and transient components is necessary to construct radiance fields with such in-the-wild photographs where existing methods require a lot of training data. We introduce SF-NeRF, aiming to disentangle those two components with only a few images given, which exploits semantic information without any supervision. The proposed method contains an occlusion filtering module that predicts the transient color and its opacity for each pixel, which enables the NeRF model to solely learn the static scene representation. This filtering module learns the transient phenomena guided by pixel-wise semantic features obtained by a trainable image encoder that can be trained across multiple scenes to learn the prior of transient objects. Furthermore, we present two techniques to prevent ambiguous decomposition and noisy results of the filtering module. We demonstrate that our method outperforms state-of-the-art novel view synthesis methods on Phototourism dataset in a few-shot setting.

cs.CV↗

arXiv2023

A Neural Pre-Conditioning Active Learning Algorithm to Reduce Label Complexity

Deep learning (DL) algorithms rely on massive amounts of labeled data. Semi-supervised learning (SSL) and active learning (AL) aim to reduce this label complexity by leveraging unlabeled data or carefully acquiring labels, respectively. In this work, we primarily focus on designing an AL algorithm but first argue for a change in how AL algorithms should be evaluated. Although unlabeled data is readily available in pool-based AL, AL algorithms are usually evaluated by measuring the increase in supervised learning (SL) performance at consecutive acquisition steps. Because this measures performance gains from both newly acquired instances and newly acquired labels, we propose to instead evaluate the label efficiency of AL algorithms by measuring the increase in SSL performance at consecutive acquisition steps. After surveying tools that can be used to this end, we propose our neural pre-conditioning (NPC) algorithm inspired by a Neural Tangent Kernel (NTK) analysis. Our algorithm incorporates the classifier's uncertainty on unlabeled data and penalizes redundant samples within candidate batches to efficiently acquire a diverse set of informative labels. Furthermore, we prove that NPC improves downstream training in the large-width regime in a manner previously observed to correlate with generalization. Comparisons with other AL algorithms show that a state-of-the-art SSL algorithm coupled with NPC can achieve high performance using very few labeled data.

cs.LG↗

arXiv2023

Robust Camera Pose Refinement for Multi-Resolution Hash Encoding

Multi-resolution hash encoding has recently been proposed to reduce the computational cost of neural renderings, such as NeRF. This method requires accurate camera poses for the neural renderings of given scenes. However, contrary to previous methods jointly optimizing camera poses and 3D scenes, the naive gradient-based camera pose refinement method using multi-resolution hash encoding severely deteriorates performance. We propose a joint optimization algorithm to calibrate the camera pose and learn a geometric representation using efficient multi-resolution hash encoding. Showing that the oscillating gradient flows of hash encoding interfere with the registration of camera poses, our method addresses the issue by utilizing smooth interpolation weighting to stabilize the gradient oscillation for the ray samplings across hash grids. Moreover, the curriculum training procedure helps to learn the level-wise hash encoding, further increasing the pose refinement. Experiments on the novel-view synthesis datasets validate that our learning frameworks achieve state-of-the-art performance and rapid convergence of neural rendering, even when initial camera poses are unknown.

cs.CV↗

arXiv2023

Domain Generalization Emerges from Dreaming

Recent studies have proven that DNNs, unlike human vision, tend to exploit texture information rather than shape. Such texture bias is one of the factors for the poor generalization performance of DNNs. We observe that the texture bias negatively affects not only in-domain generalization but also out-of-distribution generalization, i.e., Domain Generalization. Motivated by the observation, we propose a new framework to reduce the texture bias of a model by a novel optimization-based data augmentation, dubbed Stylized Dream. Our framework utilizes adaptive instance normalization (AdaIN) to augment the style of an original image yet preserve the content. We then adopt a regularization loss to predict consistent outputs between Stylized Dream and original images, which encourages the model to learn shape-based representations. Extensive experiments show that the proposed method achieves state-of-the-art performance in out-of-distribution settings on public benchmark datasets: PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet.

cs.CV↗

arXiv2022

Building a Performance Model for Deep Learning Recommendation Model Training on GPUs

We devise a performance model for GPU training of Deep Learning Recommendation Models (DLRM), whose GPU utilization is low compared to other well-optimized CV and NLP models. We show that both the device active time (the sum of kernel runtimes) but also the device idle time are important components of the overall device time. We therefore tackle them separately by (1) flexibly adopting heuristic-based and ML-based kernel performance models for operators that dominate the device active time, and (2) categorizing operator overheads into five types to determine quantitatively their contribution to the device active time. Combining these two parts, we propose a critical-path-based algorithm to predict the per-batch training time of DLRM by traversing its execution graph. We achieve less than 10% geometric mean average error (GMAE) in all kernel performance modeling, and 4.61% and 7.96% geomean errors for GPU active time and overall E2E per-batch training time prediction with overheads from individual workloads, respectively. A slight increase of 2.19% incurred in E2E prediction error with shared overheads across workloads suggests the feasibility of using shared overheads in large-scale prediction. We show that our general performance model not only achieves low prediction error on DLRM, which has highly customized configurations and is dominated by multiple factors but also yields comparable accuracy on other compute-bound ML models targeted by most previous methods. Using this performance model and graph-level data and task dependency analysis, we show our system can provide more general model-system co-design than previous methods.

cs.LG↗