arXiv subjects

Junsheng Huang

Publications and source records attributed to Junsheng Huang.

8 recordsLinked to original sources

arXiv2025

Understanding Dynamic Scenes in Ego Centric 4D Point Clouds

Understanding dynamic 4D scenes from an egocentric perspective-modeling changes in 3D spatial structure over time-is crucial for human-machine interaction, autonomous navigation, and embodied intelligence. While existing egocentric datasets contain dynamic scenes, they lack unified 4D annotations and task-driven evaluation protocols for fine-grained spatio-temporal reasoning, especially on motion of objects and human, together with their interactions. To address this gap, we introduce EgoDynamic4D, a novel QA benchmark on highly dynamic scenes, comprising RGB-D video, camera poses, globally unique instance masks, and 4D bounding boxes. We construct 927K QA pairs accompanied by explicit Chain-of-Thought (CoT), enabling verifiable, step-by-step spatio-temporal reasoning. We design 12 dynamic QA tasks covering agent motion, human-object interaction, trajectory prediction, relation understanding, and temporal-causal reasoning, with fine-grained, multidimensional metrics. To tackle these tasks, we propose an end-to-end spatio-temporal reasoning framework that unifies dynamic and static scene information, using instance-aware feature encoding, time and camera encoding, and spatially adaptive down-sampling to compress large 4D scenes into token sequences manageable by LLMs. Experiments on EgoDynamic4D show that our method consistently outperforms baselines, validating the effectiveness of multimodal temporal modeling for egocentric dynamic scene understanding.

cs.CV↗

arXiv2025

MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awareness

The hallucination of non-existent facts by LLMs is an important problem given its widespread adoption across various applications. Previous research addresses this problem by analyzing the internal parameterized knowledge boundaries to estimate confidence. However, these studies focus on the single-problem setting and have not explored the more challenging multi-problem setting, which requires accurately answering multiple questions simultaneously. We introduce a novel method for the multi-problem setting, Multiple Answers and Confidence Stepwise Tuning (MAC-Tuning), that separates the learning of answer prediction and confidence estimation during fine-tuning on instruction data. Extensive experiments demonstrate that our method outperforms baselines by up to 25\% in average precision.

cs.CL↗

arXiv2024

Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment

Safety alignment of Large Language Models (LLMs) has recently become a critical objective of model developers. In response, a growing body of work has been investigating how safety alignment can be bypassed through various jailbreaking methods, such as adversarial attacks. However, these jailbreak methods can be rather costly or involve a non-trivial amount of creativity and effort, introducing the assumption that malicious users are high-resource or sophisticated. In this paper, we study how simple random augmentations to the input prompt affect safety alignment effectiveness in state-of-the-art LLMs, such as Llama 3 and Qwen 2. We perform an in-depth evaluation of 17 different models and investigate the intersection of safety under random augmentations with multiple dimensions: augmentation type, model size, quantization, fine-tuning-based defenses, and decoding strategies (e.g., sampling temperature). We show that low-resource and unsophisticated attackers, i.e. $\textit{stochastic monkeys}$, can significantly improve their chances of bypassing alignment with just 25 random augmentations per prompt. Source code and data: https://github.com/uiuc-focal-lab/stochastic-monkeys/

cs.LG↗

arXiv2024

Few-Shot Joint Multimodal Entity-Relation Extraction via Knowledge-Enhanced Cross-modal Prompt Model

Joint Multimodal Entity-Relation Extraction (JMERE) is a challenging task that aims to extract entities and their relations from text-image pairs in social media posts. Existing methods for JMERE require large amounts of labeled data. However, gathering and annotating fine-grained multimodal data for JMERE poses significant challenges. Initially, we construct diverse and comprehensive multimodal few-shot datasets fitted to the original data distribution. To address the insufficient information in the few-shot setting, we introduce the \textbf{K}nowledge-\textbf{E}nhanced \textbf{C}ross-modal \textbf{P}rompt \textbf{M}odel (KECPM) for JMERE. This method can effectively address the problem of insufficient information in the few-shot setting by guiding a large language model to generate supplementary background knowledge. Our proposed method comprises two stages: (1) a knowledge ingestion stage that dynamically formulates prompts based on semantic similarity guide ChatGPT generating relevant knowledge and employs self-reflection to refine the knowledge; (2) a knowledge-enhanced language model stage that merges the auxiliary knowledge with the original input and utilizes a transformer-based model to align with JMERE's required output format. We extensively evaluate our approach on a few-shot dataset derived from the JMERE dataset, demonstrating its superiority over strong baselines in terms of both micro and macro F$_1$ scores. Additionally, we present qualitative analyses and case studies to elucidate the effectiveness of our model.

cs.CL↗

arXiv2024

CityCraft: A Real Crafter for 3D City Generation

City scene generation has gained significant attention in autonomous driving, smart city development, and traffic simulation. It helps enhance infrastructure planning and monitoring solutions. Existing methods have employed a two-stage process involving city layout generation, typically using Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), or Transformers, followed by neural rendering. These techniques often exhibit limited diversity and noticeable artifacts in the rendered city scenes. The rendered scenes lack variety, resembling the training images, resulting in monotonous styles. Additionally, these methods lack planning capabilities, leading to less realistic generated scenes. In this paper, we introduce CityCraft, an innovative framework designed to enhance both the diversity and quality of urban scene generation. Our approach integrates three key stages: initially, a diffusion transformer (DiT) model is deployed to generate diverse and controllable 2D city layouts. Subsequently, a Large Language Model(LLM) is utilized to strategically make land-use plans within these layouts based on user prompts and language guidelines. Based on the generated layout and city plan, we utilize the asset retrieval module and Blender for precise asset placement and scene construction. Furthermore, we contribute two new datasets to the field: 1)CityCraft-OSM dataset including 2D semantic layouts of urban areas, corresponding satellite images, and detailed annotations. 2) CityCraft-Buildings dataset, featuring thousands of diverse, high-quality 3D building assets. CityCraft achieves state-of-the-art performance in generating realistic 3D cities.

cs.CV↗

arXiv2024

Nonvolatile spin field effect transistor based on VSi2N4/Sc2CO2 multiferroic heterostructure

In this study, we present first-principles calculations that introduce a novel nonvolatile spin field-effect transistor (Spin-FET) utilizing a van der Waals multiferroic heterostructure, specifically VSi2N4/Sc2CO2. We demonstrate that inverting the ferroelectric polarization in a Sc2CO2 monolayer can effectively modulate the electronic states of a VSi2N4 monolayer, enabling a transition from half-metal to half-semiconductor. This transition significantly alters the electronic transport properties. Furthermore, we construct a Spin-FET device based on this multiferroic heterostructure and observe that the VSi2N4/Sc2CO2-based Spin-FET exhibits exceptional all-electric-controlled performance. Notably, the inversion of the Sc2CO2 ferroelectric polarization yields a substantial on-off current ratio, approximately 650\%, under a minimal bias voltage of 0.02 V. Additionally, we identify a unique spatially-separated spin-polarized transport phenomenon, wherein pure spin-up electrons transport exclusively through VSi2N4, and spin-down electrons through Sc$_2$CO$_2$. Our findings suggest a promising pathway for developing low-energy-dissipation and nonvolatile FET devices.

cond-mat.mes-hall↗

arXiv2023

CityGen: Infinite and Controllable City Layout Generation

The recent surge in interest in city layout generation underscores its significance in urban planning and smart city development. The task involves procedurally or automatically generating spatial arrangements for urban elements such as roads, buildings, water, and vegetation. Previous methods, whether procedural modeling or deep learning-based approaches like VAEs and GANs, rely on complex priors, expert guidance, or initial layouts, and often lack diversity and interactivity. In this paper, we present CityGen, an end-to-end framework for infinite, diverse, and controllable city layout generation. Our framework introduces an infinite expansion module to extend local layouts to city-scale layouts and a multi-scale refinement module to upsample and refine them. We also designed a user-friendly control scheme, allowing users to guide generation through simple sketching. Additionally, we convert the 2D layout to 3D by synthesizing a height field, facilitating downstream applications. Extensive experiments demonstrate CityGen's state-of-the-art performance across various metrics, making it suitable for a wide range of downstream applications.

cs.CV↗

arXiv2021

Exotic spintronic properties of transition-metal monolayers on graphyne

The recent discovery of two-dimensional (2D) magnetic materials which are compound of transition metal (TM) with other elements, has opened new avenues for basic research on low-dimensional magnetism and potential applications in spintronics. To further explore new 2D magnets of pure TM is thus of an interesting topic. Based on the first-principles calculations, here we propose a strategy of obtaining monolayer TM magnets, i.e., depositing TM atoms on graphyne (Gy) which has proper hexagonal hollow geometry. We find that TM monolayer with perfect hexagonal geometry can be formed on Gy. The TM monolayer exhibits a wealth of physical properties in dependence of TM species, such as ferromagnetic and antiferromagnetic ground states, as well as intriguing semimetal and half-metal characteristics. We also find that the half-metal characteristics makes the monolayer TM have great potential applications in the horizontal magnetic tunnel junction (MTJ) devices, where the tunneling magnetoresistance can reach as high as 850000%. Our results provide a new framework for obtaining 2D magnets with outstanding spintronic properties.

cond-mat.mes-hall↗