Searcharxiv⌕ Search

arXiv subjects

Jongjin Park

Publications and source records attributed to Jongjin Park.

10 recordsLinked to original sources

Stochastic control representations for extremal solutions of HJB Dirichlet problems with measurable coefficients

We study stochastic control problems for uniformly elliptic controlled diffusions on bounded domains whose coefficients are merely measurable in the state variable and whose drift is dominated by an $L_d$-function. The associated Hamilton-Jacobi-Bellman (HJB) Dirichlet problems may have more than one $L_p$-viscosity solution, so that the equation alone does not determine the value function. We show that the control problem selects the minimal solution: for all $p$ in a range $(d_0,d)$ determined by the spatial dimension, the ellipticity constant and the drift bound, the value function is the minimal $L_p$-viscosity supersolution of the HJB Dirichlet problem. In particular, every continuous $L_p$-viscosity supersolution bounds the optimal cost from above, which yields a one-sided verification theorem without regularity assumptions. We further prove the existence of optimal relaxed control rules for running costs dominated by an $L_p$-function, and the lower semicontinuity of the value function under perturbations of the coefficients. An example shows that upper semicontinuity may fail even when the coefficients are Lipschitz continuous in the control variable and the approximating coefficients are smooth, so that regularization may overestimate the optimal cost. The proof approximates arbitrary supersolutions from below by Sobolev supersolutions of HJB equations whose coefficients are modified only on sets of vanishing measure, and passes to the limit by means of Krylov estimates and martingale problem techniques.

math.OC↗

Valuation Reveals Uncertainty

This paper studies the recovery of uncertainty from dynamic sublinear valuation rules. Robust valuation assigns each payoff its worst-case expected value over a family of plausible models, thereby inducing a dynamic sublinear valuation rule. While valuation rules are observable in practice, the underlying uncertainty structure remains latent. Under suitable conditions, we show that an uncertainty structure can be recovered from an observed valuation rule and provide an explicit recovery procedure. The construction characterizes the joint uncertainty in the discount factor and the law of the state process.

q-fin.MF↗

Sublinear Markov Semigroups Beyond Uniqueness

To what extent does a local operator on smooth test functions determine a sub-Markov semigroup and its underlying structure of laws? Many existing approaches work with Feller semigroups on the space \(C_b\) of bounded continuous functions and rely on uniqueness, often established through a comparison principle for the associated evolution equation. However, even when the local operator is linear, uniqueness may fail and a \(C_b\)-Feller restriction may exclude the associated canonical semigroup. We discover that the space \(\operatorname{USC}_b\) of bounded upper semicontinuous functions offers a natural setting for this study. Within this framework, a complete description of the relationships among sublinear local operators, sublinear sub-Markov semigroups, and their underlying structures of laws is achieved without assuming uniqueness. Under suitable regularity and Lyapunov conditions, we construct the greatest semigroup compatible with a prescribed sublinear local operator and obtain a stochastic representation in terms of a Markov process under sublinear expectations. We characterize all sublinear sub-Markov semigroups dominated by this greatest semigroup in two ways: through a one-to-one correspondence with stable subfamilies of its representing laws, and through an order relation between their infinitesimal generators and the prescribed local operator. We further characterize the greatest semigroup via the greatest viscosity subsolution of the associated Hamilton--Jacobi--Bellman equation. These results clarify what information about global stochastic evolution can be recovered from local infinitesimal descriptions and what freedom remains in the absence of uniqueness. The resulting characterizations are new even when the local operator is linear.

q-fin.MF↗

SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs

Large language models (LLMs) have made significant advancements in various natural language processing tasks, including question answering (QA) tasks. While incorporating new information with the retrieval of relevant passages is a promising way to improve QA with LLMs, the existing methods often require additional fine-tuning which becomes infeasible with recent LLMs. Augmenting retrieved passages via prompting has the potential to address this limitation, but this direction has been limitedly explored. To this end, we design a simple yet effective framework to enhance open-domain QA (ODQA) with LLMs, based on the summarized retrieval (SuRe). SuRe helps LLMs predict more accurate answers for a given question, which are well-supported by the summarized retrieval that could be viewed as an explicit rationale extracted from the retrieved passages. Specifically, SuRe first constructs summaries of the retrieved passages for each of the multiple answer candidates. Then, SuRe confirms the most plausible answer from the candidate set by evaluating the validity and ranking of the generated summaries. Experimental results on diverse ODQA benchmarks demonstrate the superiority of SuRe, with improvements of up to 4.6% in exact match (EM) and 4.0% in F1 score over standard prompting approaches. SuRe also can be integrated with a broad range of retrieval methods and LLMs. Finally, the generated summaries from SuRe show additional advantages to measure the importance of retrieved passages and serve as more preferred rationales by models and humans.

cs.CL↗

Preference Transformer: Modeling Human Preferences using Transformers for RL

Preference-based reinforcement learning (RL) provides a framework to train agents using human preferences between two behaviors. However, preference-based RL has been challenging to scale since it requires a large amount of human feedback to learn a reward function aligned with human intent. In this paper, we present Preference Transformer, a neural architecture that models human preferences using transformers. Unlike prior approaches assuming human judgment is based on the Markovian rewards which contribute to the decision equally, we introduce a new preference model based on the weighted sum of non-Markovian rewards. We then design the proposed preference model using a transformer architecture that stacks causal and bidirectional self-attention layers. We demonstrate that Preference Transformer can solve a variety of control tasks using real human preferences, while prior approaches fail to work. We also show that Preference Transformer can induce a well-specified reward and attend to critical events in the trajectory by automatically capturing the temporal dependencies in human decision-making. Code is available on the project website: https://sites.google.com/view/preference-transformer.

cs.LG↗

Meta-Learning with Self-Improving Momentum Target

The idea of using a separately trained target model (or teacher) to improve the performance of the student model has been increasingly popular in various machine learning domains, and meta-learning is no exception; a recent discovery shows that utilizing task-wise target models can significantly boost the generalization performance. However, obtaining a target model for each task can be highly expensive, especially when the number of tasks for meta-learning is large. To tackle this issue, we propose a simple yet effective method, coined Self-improving Momentum Target (SiMT). SiMT generates the target model by adapting from the temporal ensemble of the meta-learner, i.e., the momentum network. This momentum network and its task-specific adaptations enjoy a favorable generalization performance, enabling self-improving of the meta-learner through knowledge distillation. Moreover, we found that perturbing parameters of the meta-learner, e.g., dropout, further stabilize this self-improving process by preventing fast convergence of the distillation loss during meta-training. Our experimental results demonstrate that SiMT brings a significant performance gain when combined with a wide range of meta-learning methods under various applications, including few-shot regression, few-shot classification, and meta-reinforcement learning. Code is available at https://github.com/jihoontack/SiMT.

cs.LG↗

OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data

Semi-supervised learning (SSL) has been a powerful strategy to incorporate few labels in learning better representations. In this paper, we focus on a practical scenario that one aims to apply SSL when unlabeled data may contain out-of-class samples - those that cannot have one-hot encoded labels from a closed-set of classes in label data, i.e., the unlabeled data is an open-set. Specifically, we introduce OpenCoS, a simple framework for handling this realistic semi-supervised learning scenario based upon a recent framework of self-supervised visual representation learning. We first observe that the out-of-class samples in the open-set unlabeled dataset can be identified effectively via self-supervised contrastive learning. Then, OpenCoS utilizes this information to overcome the failure modes in the existing state-of-the-art semi-supervised methods, by utilizing one-hot pseudo-labels and soft-labels for the identified in- and out-of-class unlabeled data, respectively. Our extensive experimental results show the effectiveness of OpenCoS under the presence of out-of-class samples, fixing up the state-of-the-art semi-supervised methods to be suitable for diverse scenarios involving open-set unlabeled data.

cs.CV↗

SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning

Preference-based reinforcement learning (RL) has shown potential for teaching agents to perform the target tasks without a costly, pre-defined reward function by learning the reward with a supervisor's preference between the two agent behaviors. However, preference-based learning often requires a large amount of human feedback, making it difficult to apply this approach to various applications. This data-efficiency problem, on the other hand, has been typically addressed by using unlabeled samples or data augmentation techniques in the context of supervised learning. Motivated by the recent success of these approaches, we present SURF, a semi-supervised reward learning framework that utilizes a large amount of unlabeled samples with data augmentation. In order to leverage unlabeled samples for reward learning, we infer pseudo-labels of the unlabeled samples based on the confidence of the preference predictor. To further improve the label-efficiency of reward learning, we introduce a new data augmentation that temporally crops consecutive subsequences from the original behaviors. Our experiments demonstrate that our approach significantly improves the feedback-efficiency of the state-of-the-art preference-based method on a variety of locomotion and robotic manipulation tasks.

cs.LG↗

Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning

Behavioral cloning has proven to be effective for learning sequential decision-making policies from expert demonstrations. However, behavioral cloning often suffers from the causal confusion problem where a policy relies on the noticeable effect of expert actions due to the strong correlation but not the cause we desire. This paper presents Object-aware REgularizatiOn (OREO), a simple technique that regularizes an imitation policy in an object-aware manner. Our main idea is to encourage a policy to uniformly attend to all semantic objects, in order to prevent the policy from exploiting nuisance variables strongly correlated with expert actions. To this end, we introduce a two-stage approach: (a) we extract semantic objects from images by utilizing discrete codes from a vector-quantized variational autoencoder, and (b) we randomly drop the units that share the same discrete code together, i.e., masking out semantic objects. Our experiments demonstrate that OREO significantly improves the performance of behavioral cloning, outperforming various other regularization and causality-based methods on a variety of Atari environments and a self-driving CARLA environment. We also show that our method even outperforms inverse reinforcement learning methods trained with a considerable amount of environment interaction.

cs.LG↗

Regularizing Class-wise Predictions via Self-knowledge Distillation

Deep neural networks with millions of parameters may suffer from poor generalization due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label during training. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network (i.e., a self-knowledge distillation) by forcing it to produce more meaningful and consistent predictions in a class-wise manner. Consequently, it mitigates overconfident predictions and reduces intra-class variations. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve not only the generalization ability but also the calibration performance of modern convolutional neural networks.

cs.LG↗