arXiv subjects

Guojun Chen

Publications and source records attributed to Guojun Chen.

16 recordsLinked to original sources

arXiv2026

TypeGo: An OS Runtime for Embodied Agents

Large language models (LLMs) can plan behavior for embodied agents from natural language, but treating the LLM as a request/response oracle on the critical path is fundamentally at odds with real-time control and concurrent goals. We argue for an operating-system-style runtime for embodied agents, and instantiate this idea in an early prototype, TypeGo. TypeGo structures LLM-based planning as asynchronous loops at multiple timescales that overlap with execution, and manages the agent's physical body like an OS manages hardware: the Skill Kernel arbitrates typed physical subsystems among concurrent per-task processes, a scheduler preempts them and resumes or replaces each by source, and speculative skill streaming hides LLM latency behind ongoing motion, while a fast first-action path yields visible feedback within a second. Users program behavior through natural language prescriptions that TypeGo dispatches to the LLM-based planners or compiles into low-latency interrupt handlers. Our prototype of Kalos, a Unitree Go2 quadruped, provides preliminary evidence for the design: in our current task suite, it cuts per-step delay by 50% over step-by-step planning and time-to-first-action by 73% over monolithic planning, while admitting concurrent tasks at low scheduling overhead.

cs.SE↗

arXiv2025

VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image

We propose VASA-3D, an audio-driven, single-shot 3D head avatar generator. This research tackles two major challenges: capturing the subtle expression details present in real human faces, and reconstructing an intricate 3D head avatar from a single portrait image. To accurately model expression details, VASA-3D leverages the motion latent of VASA-1, a method that yields exceptional realism and vividness in 2D talking heads. A critical element of our work is translating this motion latent to 3D, which is accomplished by devising a 3D head model that is conditioned on the motion latent. Customization of this model to a single image is achieved through an optimization framework that employs numerous video frames of the reference head synthesized from the input image. The optimization takes various training losses robust to artifacts and limited pose coverage in the generated training data. Our experiment shows that VASA-3D produces realistic 3D talking heads that cannot be achieved by prior art, and it supports the online generation of 512x512 free-viewpoint videos at up to 75 FPS, facilitating more immersive engagements with lifelike 3D avatars.

cs.CV↗

arXiv2024

TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applications

Large Language Models (LLMs) such as GPT-4 and Llama3 can already comprehend complex commands and process diverse tasks. This advancement facilitates their application in controlling drones and robots for various tasks. However, existing LLM serving systems typically employ a first-come, first-served (FCFS) batching mechanism, which fails to address the time-sensitive requirements of robotic applications. To address it, this paper proposes a new system named TimelyLLM serving multiple robotic agents with time-sensitive requests. TimelyLLM introduces novel mechanisms of segmented generation and scheduling that optimally leverage redundancy between robot plan generation and execution phases. We report an implementation of TimelyLLM on a widely-used LLM serving framework and evaluate it on a range of robotic applications. Our evaluation shows that TimelyLLM improves the time utility up to 1.97x, and reduces the overall waiting time by 84%.

cs.RO↗

arXiv2024

Cold plasma with zirconia nanoparticles for lung cancer via TGF-\b{eta} signaling pathway

Despite advancements in lung cancer therapy, the prognosis for advanced or metastatic patients remains poor, yet many patients eventually develop resistance to standard treatments leading to disease progression and poor survival. Here, we described a combination of CAP and nanoparticles (ZrO2 NPs (zirconium oxide nanoparticle) and 3Y-TZP NPs (3% mol Yttria Tetragonal Zirconia Polycrystal Nanoparticle)) for lung cancer therapy. We found that ZrO2 NPs caused obvious damage to the inside of the lung cancer cells. CAP and ZrO2 NPs mainly affected the mitochondria function, leading to a decrease in mitochondrial membrane potential and ATP levels, and causing endoplasmic reticulum stress and cell nucleus internal DNA damage, etc. CAP combined with ZrO2 NPs (CAP@ZrO2) induced lung cancer cell apoptosis by activating the TGF-\b{eta} pathway. CAP@ZrO2 offers a new therapy for the clinical treatment of lung cancer.

physics.bio-ph↗

arXiv2024

Prompt Cache: Modular Attention Reuse for Low-Latency Inference

We present Prompt Cache, an approach for accelerating inference for large language models (LLM) by reusing attention states across different LLM prompts. Many input prompts have overlapping text segments, such as system messages, prompt templates, and documents provided for context. Our key insight is that by precomputing and storing the attention states of these frequently occurring text segments on the inference server, we can efficiently reuse them when these segments appear in user prompts. Prompt Cache employs a schema to explicitly define such reusable text segments, called prompt modules. The schema ensures positional accuracy during attention state reuse and provides users with an interface to access cached states in their prompt. Using a prototype implementation, we evaluate Prompt Cache across several LLMs. We show that Prompt Cache significantly reduce latency in time-to-first-token, especially for longer prompts such as document-based question answering and recommendations. The improvements range from 8x for GPU-based inference to 60x for CPU-based inference, all while maintaining output accuracy and without the need for model parameter modifications.

cs.CL↗

arXiv2024

VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

We introduce VASA, a framework for generating lifelike talking faces with appealing visual affective skills (VAS) given a single static image and a speech audio clip. Our premiere model, VASA-1, is capable of not only generating lip movements that are exquisitely synchronized with the audio, but also producing a large spectrum of facial nuances and natural head motions that contribute to the perception of authenticity and liveliness. The core innovations include a holistic facial dynamics and head movement generation model that works in a face latent space, and the development of such an expressive and disentangled face latent space using videos. Through extensive experiments including evaluation on a set of new metrics, we show that our method significantly outperforms previous methods along various dimensions comprehensively. Our method not only delivers high video quality with realistic facial and head dynamics but also supports the online generation of 512x512 videos at up to 40 FPS with negligible starting latency. It paves the way for real-time engagements with lifelike avatars that emulate human conversational behaviors.

cs.CV↗

arXiv2023

TypeFly: Flying Drones with Large Language Model

Recent advancements in robot control using large language models (LLMs) have demonstrated significant potential, primarily due to LLMs' capabilities to understand natural language commands and generate executable plans in various languages. However, in real-time and interactive applications involving mobile robots, particularly drones, the sequential token generation process inherent to LLMs introduces substantial latency, i.e. response time, in control plan generation. In this paper, we present a system called ChatFly that tackles this problem using a combination of a novel programming language called MiniSpec and its runtime to reduce the plan generation time and drone response time. That is, instead of asking an LLM to write a program (robotic plan) in the popular but verbose Python, ChatFly gets it to do it in MiniSpec specially designed for token efficiency and stream interpretation. Using a set of challenging drone tasks, we show that design choices made by ChatFly can reduce up to 62% response time and provide a more consistent user experience, enabling responsive and intelligent LLM-based drone control with efficient completion.

cs.RO↗

arXiv2023

Relighting Neural Radiance Fields with Shadow and Highlight Hints

This paper presents a novel neural implicit radiance representation for free viewpoint relighting from a small set of unstructured photographs of an object lit by a moving point light source different from the view position. We express the shape as a signed distance function modeled by a multi layer perceptron. In contrast to prior relightable implicit neural representations, we do not disentangle the different reflectance components, but model both the local and global reflectance at each point by a second multi layer perceptron that, in addition, to density features, the current position, the normal (from the signed distace function), view direction, and light position, also takes shadow and highlight hints to aid the network in modeling the corresponding high frequency light transport effects. These hints are provided as a suggestion, and we leave it up to the network to decide how to incorporate these in the final relit result. We demonstrate and validate our neural implicit representation on synthetic and real scenes exhibiting a wide variety of shapes, material properties, and global illumination light transport.

cs.CV↗

arXiv2023

Air cold atmospheric plasma with patterns for anaplastic squamous cell carcinoma treatment

In recent years, cold atmospheric plasma (CAP) using inert gas has been successfully applied for biomedicine, such as sterilization, wound healing, skin diseases, and tumor treatment. Here, we reported air cold atmospheric plasma with three different patterns (I. Non: basic square grid structure; II. Square: basic square grid structure + square node; III. Circle: basic square grid structure + circle node) for anaplastic squamous cell carcinoma treatment (VX2 cell line). Various plasma diagnostic techniques were applied to evaluate the physics of air CAP with patterns such as discharge voltage, plasma initial generating process, plasma temperature, and optical emission spectroscopy (OES). The direct effects of air CAP with patterns on anaplastic squamous cell carcinoma treatment (VX2 cell line) were investigated in vitro. We also studied the ROS (reactive oxygen species) and RNS (reactive nitrogen species) generation in cultured media released from VX2 cells after the treatment of air CAP with patterns. The results showed that the air CAP with circle-pattern generated more active substances during at 60s treatment time, which resulted in a higher death rate of VX2 cells. These initial observations establish the air CAP with patterns as potential clinical applications for cancer therapy.

physics.med-ph↗

arXiv2023

Computing the Rate-Distortion Function of Gray-Wyner System

In this paper, the rate-distortion theory of the Gray-Wyner lossy source coding system is investigated. For the case of jointly Gaussian distributed sources, we establish an expression for the rate-distortion function under the constraint of quadratic distortion. Using the proposed rate-distortion function, any corner point on the rate-distortion region can be conveniently calculated. We take Wyner's common information as an example and provide a general and simple method to solve this problem. Through the analysis of the rate-distortion function, the rate on each layer and covariance matrix of auxiliary random variables and the sources are also presented in this paper.

cs.IT↗

arXiv2021

VirtualCube: An Immersive 3D Video Communication System

The VirtualCube system is a 3D video conference system that attempts to overcome some limitations of conventional technologies. The key ingredient is VirtualCube, an abstract representation of a real-world cubicle instrumented with RGBD cameras for capturing the 3D geometry and texture of a user. We design VirtualCube so that the task of data capturing is standardized and significantly simplified, and everything can be built using off-the-shelf hardware. We use VirtualCubes as the basic building blocks of a virtual conferencing environment, and we provide each VirtualCube user with a surrounding display showing life-size videos of remote participants. To achieve real-time rendering of remote participants, we develop the V-Cube View algorithm, which uses multi-view stereo for more accurate depth estimation and Lumi-Net rendering for better rendering quality. The VirtualCube system correctly preserves the mutual eye gaze between participants, allowing them to establish eye contact and be aware of who is visually paying attention to them. The system also allows a participant to have side discussions with remote participants as if they were in the same room. Finally, the system sheds lights on how to support the shared space of work items (e.g., documents and applications) and track the visual attention of participants to work items.

cs.CV↗

arXiv2021

New Proofs of Extremal Inequalities With Applications

The extremal inequality approach plays a key role in network information theory problems. In this paper, we propose a novel monotone path construction in product probability space. The optimality of Gaussian distribution is then established by standard perturbation arguments. The proofs of Liu-Viswanath extremal and vector Generalization of Costa's entropy power inequality are illustrated into the unified framework. As applications, capacity region of the multiple-input multiple-output (MIMO) Gaussian broadcast channel and rate-distortion-equivocation function of the vector Gaussian secure source coding are revisited through our proposed extremal inequality approach.

cs.IT↗

arXiv2021

POD: A Smartphone That Flies

We present POD, a smartphone that flies, as a new way to achieve hands-free, eyes-up mobile computing. Unlike existing drone-carried user interfaces, POD features a smartphone-sized display and the computing and sensing power of a modern smartphone. We share our experience in building a prototype of POD, discuss the technical challenges facing it, and describe early results toward addressing them.

cs.HC↗

arXiv2020

Object-based Illumination Estimation with Rendering-aware Neural Networks

We present a scheme for fast environment light estimation from the RGBD appearance of individual objects and their local image areas. Conventional inverse rendering is too computationally demanding for real-time applications, and the performance of purely learning-based techniques may be limited by the meager input data available from individual objects. To address these issues, we propose an approach that takes advantage of physical principles from inverse rendering to constrain the solution, while also utilizing neural networks to expedite the more computationally expensive portions of its processing, to increase robustness to noisy input data as well as to improve temporal and spatial stability. This results in a rendering-aware system that estimates the local illumination distribution at an object with high accuracy and in real time. With the estimated lighting, virtual objects can be rendered in AR scenarios with shading that is consistent to the real scene, leading to improved realism.

cs.CV↗

arXiv2019

An online supervised learning algorithm based on triple spikes for spiking neural networks

Using precise times of every spike, spiking supervised learning has more effects on complex spatial-temporal pattern than supervised learning only through neuronal firing rates. The purpose of spiking supervised learning after spatial-temporal encoding is to emit desired spike trains with precise times. Existing algorithms of spiking supervised learning have excellent performances, but mechanisms of them still have some problems, such as the limitation of neuronal types and complex computation. Based on an online regulative mechanism of biological synapses, this paper proposes an online supervised learning algorithm of multiple spike trains for spiking neural networks. The proposed algorithm with a spatial-temporal transformation can make a simple direct regulation of synaptic weights as soon as firing time of an output spike is obtained. Besides, it is also not restricted by types of spiking neuron models. Relationship among desired output, actual output and input spike trains is firstly analyzed and synthesized to simply select a unit of pair-spike for a direct regulation. And then a computational method is constructed based on simple triple spikes using this direct regulation. Compared with other learning algorithms, results of experiments show that proposed algorithm has higher learning accuracy and efficiency.

cs.NE↗

arXiv2017

Robust Keyframe-based Dense SLAM with an RGB-D Camera

In this paper, we present RKD-SLAM, a robust keyframe-based dense SLAM approach for an RGB-D camera that can robustly handle fast motion and dense loop closure, and run without time limitation in a moderate size scene. It not only can be used to scan high-quality 3D models, but also can satisfy the demand of VR and AR applications. First, we combine color and depth information to construct a very fast keyframe-based tracking method on a CPU, which can work robustly in challenging cases (e.g.~fast camera motion and complex loops). For reducing accumulation error, we also introduce a very efficient incremental bundle adjustment (BA) algorithm, which can greatly save unnecessary computation and perform local and global BA in a unified optimization framework. An efficient keyframe-based depth representation and fusion method is proposed to generate and timely update the dense 3D surface with online correction according to the refined camera poses of keyframes through BA. The experimental results and comparisons on a variety of challenging datasets and TUM RGB-D benchmark demonstrate the effectiveness of the proposed system.

cs.CV↗