arXiv subjects

Jae Sung Park

Publications and source records attributed to Jae Sung Park.

At least 37 records · Page 2Linked to original sources

arXiv2023

ArK: Augmented Reality with Knowledge Interactive Emergent Ability

Despite the growing adoption of mixed reality and interactive AI agents, it remains challenging for these systems to generate high quality 2D/3D scenes in unseen environments. The common practice requires deploying an AI agent to collect large amounts of data for model training for every new task. This process is costly, or even impossible, for many domains. In this study, we develop an infinite agent that learns to transfer knowledge memory from general foundation models (e.g. GPT4, DALLE) to novel domains or scenarios for scene understanding and generation in the physical or virtual world. The heart of our approach is an emerging mechanism, dubbed Augmented Reality with Knowledge Inference Interaction (ArK), which leverages knowledge-memory to generate scenes in unseen physical world and virtual reality environments. The knowledge interactive emergent ability (Figure 1) is demonstrated as the observation learns i) micro-action of cross-modality: in multi-modality models to collect a large amount of relevant knowledge memory data for each interaction task (e.g., unseen scene understanding) from the physical reality; and ii) macro-behavior of reality-agnostic: in mix-reality environments to improve interactions that tailor to different characterized roles, target variables, collaborative information, and so on. We validate the effectiveness of ArK on the scene generation and editing tasks. We show that our ArK approach, combined with large foundation models, significantly improves the quality of generated 2D/3D scenes, compared to baselines, demonstrating the potential benefit of incorporating ArK in generative AI for applications such as metaverse and gaming simulation.

cs.CV↗

arXiv2022

The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning

Humans have remarkable capacity to reason abductively and hypothesize about what lies beyond the literal content of an image. By identifying concrete visual clues scattered throughout a scene, we almost can't help but draw probable inferences beyond the literal scene based on our everyday experience and knowledge about the world. For example, if we see a "20 mph" sign alongside a road, we might assume the street sits in a residential area (rather than on a highway), even if no houses are pictured. Can machines perform similar visual reasoning? We present Sherlock, an annotated corpus of 103K images for testing machine capacity for abductive reasoning beyond literal image contents. We adopt a free-viewing paradigm: participants first observe and identify salient clues within images (e.g., objects, actions) and then provide a plausible inference about the scene, given the clue. In total, we collect 363K (clue, inference) pairs, which form a first-of-its-kind abductive visual reasoning dataset. Using our corpus, we test three complementary axes of abductive reasoning. We evaluate the capacity of models to: i) retrieve relevant inferences from a large candidate corpus; ii) localize evidence for inferences via bounding boxes, and iii) compare plausible inferences to match human judgments on a newly-collected diagnostic corpus of 19K Likert-scale judgments. While we find that fine-tuning CLIP-RN50x64 with a multitask objective outperforms strong baselines, significant headroom exists between model performance and human agreement. Data, models, and leaderboard available at http://visualabduction.com/

cs.CV↗

arXiv2021

On the underlying drag-reduction mechanisms of flow-control strategies in a transitional channel flow: temporal approach

The underlying mechanisms of three different flow-control strategies on drag reduction in a channel flow are investigated by direct numerical simulations at friction Reynolds numbers ranging from 65 to 85. These strategies include the addition of long-chain polymers, the incorporation of slip surfaces, and the application of an external body force. While it has been believed that such methods lead to a skin-friction reduction by controlling near-wall flow structures, the underlying mechanisms at play are still not as clear. In this study, a temporal analysis is employed to elucidate underlying drag-reduction mechanisms among these methods. The analysis is based on the lifetime of intermittent phases represented by the active and hibernating phases of a minimal turbulent channel flow (Xi & Graham, Phy. Rev. Lett. 2010). At a similar amount of drag reduction, the polymer and slip methods show a similar mechanism, while the body force method is different. The polymers and slip surfaces cause hibernating phases to happen more frequently, while the duration of active phases is decreased. However, the body forces cause hibernating phases to happen less frequently but prolong its duration to achieve a comparable amount of drag reduction. A possible mechanism behind the body force method is associated with its unique roller-like vortical structures formed near the wall. These structures appear to prevent interactions between inner and outer regions by which hibernating phases are prolonged. It should motivate adaptive flow-control strategies to exploit the distinct underlying mechanisms for robust control of turbulent drag at low Reynolds numbers.

physics.flu-dyn↗

arXiv2021

MERLOT: Multimodal Neural Script Knowledge Models

As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future. We introduce MERLOT, a model that learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech -- in an entirely label-free, self-supervised manner. By pretraining with a mix of both frame-level (spatial) and video-level (temporal) objectives, our model not only learns to match images to temporally corresponding words, but also to contextualize what is happening globally over time. As a result, MERLOT exhibits strong out-of-the-box representations of temporal commonsense, and achieves state-of-the-art performance on 12 different video QA datasets when finetuned. It also transfers well to the world of static images, allowing models to reason about the dynamic context behind visual scenes. On Visual Commonsense Reasoning, MERLOT answers questions correctly with 80.6% accuracy, outperforming state-of-the-art models of similar size by over 3%, even those that make heavy use of auxiliary supervised data (like object bounding boxes). Ablation analyses demonstrate the complementary importance of: 1) training on videos versus static images; 2) scaling the magnitude and diversity of the pretraining video corpus; and 3) using diverse objectives that encourage full-stack multimodal reasoning, from the recognition to cognition level.

cs.CV↗

arXiv2021

LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codes

Learning binary representations of instances and classes is a classical problem with several high potential applications. In modern settings, the compression of high-dimensional neural representations to low-dimensional binary codes is a challenging task and often require large bit-codes to be accurate. In this work, we propose a novel method for Learning Low-dimensional binary Codes (LLC) for instances as well as classes. Our method does not require any side-information, like annotated attributes or label meta-data, and learns extremely low-dimensional binary codes (~20 bits for ImageNet-1K). The learnt codes are super-efficient while still ensuring nearly optimal classification accuracy for ResNet50 on ImageNet-1K. We demonstrate that the learnt codes capture intrinsically important features in the data, by discovering an intuitive taxonomy over classes. We further quantitatively measure the quality of our codes by applying it to the efficient image retrieval as well as out-of-distribution (OOD) detection problems. For ImageNet-100 retrieval problem, our learnt binary codes outperform 16 bit HashNet using only 10 bits and also are as accurate as 10 dimensional real representations. Finally, our learnt binary codes can perform OOD detection, out-of-the-box, as accurately as a baseline that needs ~3000 samples to tune its threshold, while we require none. Code is open-sourced at https://github.com/RAIVNLab/LLC.

cs.LG↗

arXiv2021

Exact coherent structures and phase space geometry of pre-turbulent 2D active nematic channel flow

Confined active nematics exhibit rich dynamical behavior, including spontaneous flows, periodic defect dynamics, and chaotic `active turbulence'. Here, we study these phenomena using the framework of Exact Coherent Structures, which has been successful in characterizing the routes to high Reynolds number turbulence of passive fluids. Exact Coherent Structures are stationary, periodic, quasiperiodic, or traveling wave solutions of the hydrodynamic equations that, together with their invariant manifolds, serve as an organizing template of the dynamics. We compute the dominant Exact Coherent Structures and connecting orbits in a pre-turbulent active nematic channel flow, which enables a fully nonlinear but highly reduced order description in terms of a directed graph. Using this reduced representation, we compute instantaneous perturbations that switch the system between disparate spatiotemporal states occupying distant regions of the infinite dimensional phase space. Our results lay the groundwork for a systematic means of understanding and controlling active nematic flows in the moderate to high activity regime.

cond-mat.soft↗

arXiv2020

The zero-shear-rate limiting rheological behaviors of ideally conductive particles suspended in concentrated dispersions under an electric field

The rheological behaviors of suspension of ideally conductive particles in an electric field are studied using large-scale numerical simulations in the limit of zero-shear-rate flow. Under the action of an electric field, the particles undergo the nonlinear electrokinetic phenomenon termed as dipolophoresis, which is the combination of dielectrophoresis and induced-charge electrophoresis. For ideally conductive particles, the dynamics of the suspension are primarily controlled by induced-charge electrophoresis. To characterize the rheological properties of the suspension, the particle stress tensor and particle pressure are calculated in a range of volume fraction up to almost random close packing. The particle normal stress and particle pressure are shown to behave non-monotonically with volume fraction, especially in concentrated regimes. In particular, the particle pressure is positive for volume fraction up to 30\%, after which it becomes negative, indicating a change in the nature of the particle pressure. The microstructure expressed by pair distribution function and suspension entropy is also evaluated. Visible variations in the local microstructure seem to correlate with the non-monotonic variation in the particle normal stresses and particle pressure. These non-monotonic behaviors are also correlated with the change in the dominant mechanism of particle pairing dynamics observed in our recent study [Mirfendereski \& Park, J. Fluid Mech. \textbf{875}, R3 (2019)]. Lastly, the effects of confinement on the particle stress and particle pressure are investigated. It is found that the particle pressure changes its nature very quickly at high volume fractions as the level of confinement increases. This study should motivate control strategies to fully exploit the distinct changing nature of the pressure for rheological manipulation of such suspension system.

cond-mat.soft↗

arXiv2020

Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs

Natural language rationales could provide intuitive, higher-level explanations that are easily understandable by humans, complementing the more broadly studied lower-level explanations based on gradients or attention weights. We present the first study focused on generating natural language rationales across several complex visual reasoning tasks: visual commonsense reasoning, visual-textual entailment, and visual question answering. The key challenge of accurate rationalization is comprehensive image understanding at all levels: not just their explicit content at the pixel level, but their contextual contents at the semantic and pragmatic levels. We present Rationale^VT Transformer, an integrated model that learns to generate free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. Our experiments show that the base pretrained language model benefits from visual adaptation and that free-text rationalization is a promising research direction to complement model interpretability for complex visual-textual reasoning tasks.

cs.CL↗

arXiv2020

Identity-Aware Multi-Sentence Video Description

Standard video and movie description tasks abstract away from person identities, thus failing to link identities across sentences. We propose a multi-sentence Identity-Aware Video Description task, which overcomes this limitation and requires to re-identify persons locally within a set of consecutive clips. We introduce an auxiliary task of Fill-in the Identity, that aims to predict persons' IDs consistently within a set of clips, when the video descriptions are given. Our proposed approach to this task leverages a Transformer architecture allowing for coherent joint prediction of multiple IDs. One of the key components is a gender-aware textual representation as well an additional gender prediction objective in the main model. This auxiliary task allows us to propose a two-stage approach to Identity-Aware Video Description. We first generate multi-sentence video descriptions, and then apply our Fill-in the Identity model to establish links between the predicted person entities. To be able to tackle both tasks, we augment the Large Scale Movie Description Challenge (LSMDC) benchmark with new annotations suited for our problem statement. Experiments show that our proposed Fill-in the Identity model is superior to several baselines and recent works, and allows us to generate descriptions with locally re-identified people.

cs.CV↗

arXiv2020

VisualCOMET: Reasoning about the Dynamic Context of a Still Image

Even from a single frame of a still image, people can reason about the dynamic story of the image before, after, and beyond the frame. For example, given an image of a man struggling to stay afloat in water, we can reason that the man fell into the water sometime in the past, the intent of that man at the moment is to stay alive, and he will need help in the near future or else he will get washed away. We propose VisualComet, the novel framework of visual commonsense reasoning tasks to predict events that might have happened before, events that might happen next, and the intents of the people at present. To support research toward visual commonsense reasoning, we introduce the first large-scale repository of Visual Commonsense Graphs that consists of over 1.4 million textual descriptions of visual commonsense inferences carefully annotated over a diverse set of 60,000 images, each paired with short video summaries of before and after. In addition, we provide person-grounding (i.e., co-reference links) between people appearing in the image and people mentioned in the textual commonsense descriptions, allowing for tighter integration between images and text. We establish strong baseline performances on this task and demonstrate that integration between visual and textual commonsense reasoning is the key and wins over non-integrative alternatives.

cs.CV↗

arXiv2020

LSTM-based Anomaly Detection for Non-linear Dynamical System

Anomaly detection for non-linear dynamical system plays an important role in ensuring the system stability. However, it is usually complex and has to be solved by large-scale simulation which requires extensive computing resources. In this paper, we propose a novel anomaly detection scheme in non-linear dynamical system based on Long Short-Term Memory (LSTM) to capture complex temporal changes of the time sequence and make multi-step predictions. Specifically, we first present the framework of LSTM-based anomaly detection in non-linear dynamical system, including data preprocessing, multi-step prediction and anomaly detection. According to the prediction requirement, two types of training modes are explored in multi-step prediction, where samples in a wall shear stress dataset are collected by an adaptive sliding window. On the basis of the multi-step prediction result, a Local Average with Adaptive Parameters (LAAP) algorithm is proposed to extract local numerical features of the time sequence and estimate the upcoming anomaly. The experimental results show that our proposed multi-step prediction method can achieve a higher prediction accuracy than traditional method in wall shear stress dataset, and the LAAP algorithm performs better than the absolute value-based method in anomaly detection task.

eess.SP↗

arXiv2020

HMPO: Human Motion Prediction in Occluded Environments for Safe Motion Planning

We present a novel approach to generate collision-free trajectories for a robot operating in close proximity with a human obstacle in an occluded environment. The self-occlusions of the robot can significantly reduce the accuracy of human motion prediction, and we present a novel deep learning-based prediction algorithm. Our formulation uses CNNs and LSTMs and we augment human-action datasets with synthetically generated occlusion information for training. We also present an occlusion-aware planner that uses our motion prediction algorithm to compute collision-free trajectories. We highlight performance of the overall approach (HMPO) in complex scenarios and observe upto 68% performance improvement in motion prediction accuracy, and 38% improvement in terms of error distance between the ground-truth and the predicted human joint positions.

cs.RO↗

arXiv2020

Dynamics of laminar and transitional flows over slip surfaces: effects on the laminar-turbulent separatrix

The effect of slip surfaces on the laminar-turbulent separatrix of plane Poiseuille flow is studied by direct numerical simulation. Turbulence lifetimes, the likelihood that turbulence is sustained, is investigated for transitional flows with various slip lengths. Slip surfaces decrease the likelihood of sustained turbulence compared to the no-slip case, and likelihood is further decreased as slip length is increased. A deterministic analysis of the effects of slip surfaces on transition to turbulence is performed using nonlinear traveling wave solutions to the Navier-Stokes equations, also known as exact coherent solutions. Two solution families, dubbed P3 and P4, are used since their lower-branch solutions are embedded on the boundary of the basin of attraction of laminar and turbulent flows (Park & Graham 2015). Additionally, they exhibit distinct flow structures -- the P3 and P4 are denoted as core mode and critical layer mode, respectively. Distinct effects of slip surfaces on the solutions are observed by the skin friction evolution, linear growth rate, and phase-space projection of transitional trajectories. The slip surface modifies transition dynamics little for the core mode, but considerably for the critical layer mode. Most importantly, the slip surface promotes different transition dynamics -- early and bypass-like transition for the core mode and delayed and H-/K-type-like transition for the critical layer mode. Based on spatiotemporal and quadrant analyses, it is found that slip surfaces promote the prevalence of strong wall-toward motions (sweep-like events) near vortex cores close to the channel centre, inducing an early transition, while sustained ejection events are present in the region of the $Λ$-shaped vortex cores close to the critical layer, resulting in a delayed transition.

physics.flu-dyn↗

arXiv2019

Efficient Probabilistic Collision Detection for Non-Gaussian Noise Distributions

We present an efficient algorithm to compute tight upper bounds of collision probability between two objects with positional uncertainties, whose error distributions are represented with non-Gaussian forms. Our approach can handle noisy datasets from depth sensors, whose distributions may correspond to Truncated Gaussian, Weighted Samples, or Truncated Gaussian Mixture Model. We derive tight probability bounds for convex shapes and extend them to non-convex shapes using hierarchical representations. We highlight the benefits of our approach over prior probabilistic collision detection algorithms in terms of tighter bounds ($10$x) and improved running time ($3$x). Moreover, we use our tight bounds to design an efficient and accurate motion planning algorithm for a 7-DOF robot arm operating in tight scenarios with sensor and motion uncertainties.

cs.RO↗

arXiv2019

Dipolophoresis in concentrated suspensions of ideally polarizable spheres

The dynamics of ideally polarizable spherical particles in concentrated suspensions under the effects of nonlinear electrokinetic phenomena is analysed using large-scale numerical simulations. Particles are assumed to carry no net charge and considered to undergo the combination of dielectrophoresis and induced-charge electrophoresis termed dipolophoresis. Chaotic motion and resulting hydrodynamic diffusion are known to be driven by the induced-charge electrophoresis, which dominates the dielectrophoresis. Up to a volume fraction $ϕ\approx 35\%$, the particle dynamics seems to be hindered by the increase in the magnitude of excluded volume interactions with concentration. However, a non-trivial suspension behaviour is observed in concentrated regimes, where the hydrodynamic diffusivity starts to increase with a volume fraction at $ϕ\approx 35\%$ before reaching a local maximum and then drastically decreases as approaching random close packing. Similar non-trivial behaviours are observed in the particle velocity and number-density fluctuations around volume fractions that the non-trivial behaviour of the hydrodynamic diffusion is observed. We explain these non-trivial behaviours as a consequence of particle contacts, which are related to the dominant mechanism of particle pairings. The particle contacts are classified into attractive and repulsive classes by the nature of contacts, and in particular, the strong repulsive contact becomes predominant at $ϕ> 20\%$. Moreover, this transition is visible in the pair distribution functions, which also reveal the change in the suspension microstructure in concentrated regimes. It appears that strong and massive repulsive contacts along the direction perpendicular to an electric field promote the non-trivial suspension behaviours observed in concentrated regimes.

cond-mat.soft↗

arXiv2019

Joint High Dynamic Range Imaging and Super-Resolution from a Single Image

This paper presents a new framework for jointly enhancing the resolution and the dynamic range of an image, i.e., simultaneous super-resolution (SR) and high dynamic range imaging (HDRI), based on a convolutional neural network (CNN). From the common trends of both tasks, we train a CNN for the joint HDRI and SR by focusing on the reconstruction of high-frequency details. Specifically, the high-frequency component in our work is the reflectance component according to the Retinex-based image decomposition, and only the reflectance component is manipulated by the CNN while another component (illumination) is processed in a conventional way. In training the CNN, we devise an appropriate loss function that contributes to the naturalness quality of resulting images. Experiments show that our algorithm outperforms the cascade implementation of CNN-based SR and HDRI.

eess.IV↗

arXiv2019

Adversarial Inference for Multi-Sentence Video Description

While significant progress has been made in the image captioning task, video description is still in its infancy due to the complex nature of video data. Generating multi-sentence descriptions for long videos is even more challenging. Among the main issues are the fluency and coherence of the generated descriptions, and their relevance to the video. Recently, reinforcement and adversarial learning based methods have been explored to improve the image captioning models; however, both types of methods suffer from a number of issues, e.g. poor readability and high redundancy for RL and stability issues for GANs. In this work, we instead propose to apply adversarial techniques during inference, designing a discriminator which encourages better multi-sentence video description. In addition, we find that a multi-discriminator "hybrid" design, where each discriminator targets one aspect of a description, leads to the best results. Specifically, we decouple the discriminator to evaluate on three criteria: 1) visual relevance to the video, 2) language diversity and fluency, and 3) coherence across sentences. Our approach results in more accurate, diverse, and coherent multi-sentence video descriptions, as shown by automatic as well as human evaluation on the popular ActivityNet Captions dataset.

cs.CV↗

arXiv2018

Efficient Generation of Motion Plans from Attribute-Based Natural Language Instructions Using Dynamic Constraint Mapping

We present an algorithm for combining natural language processing (NLP) and fast robot motion planning to automatically generate robot movements. Our formulation uses a novel concept called Dynamic Constraint Mapping to transform complex, attribute-based natural language instructions into appropriate cost functions and parametric constraints for optimization-based motion planning. We generate a factor graph from natural language instructions called the Dynamic Grounding Graph (DGG), which takes latent parameters into account. The coefficients of this factor graph are learned based on conditional random fields (CRFs) and are used to dynamically generate the constraints for motion planning. We map the cost function directly to the motion parameters of the planner and compute smooth trajectories in dynamic scenes. We highlight the performance of our approach in a simulated environment and via a human interacting with a 7-DOF Fetch robot using intricate language commands including negation, orientation specification, and distance constraints.

cs.RO↗