arXiv subjects

Xin Zeng

Publications and source records attributed to Xin Zeng.

32 records · Page 2Linked to original sources

arXiv2024

GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents

Existing gesture interfaces only work with a fixed set of gestures defined either by interface designers or by users themselves, which introduces learning or demonstration efforts that diminish their naturalness. Humans, on the other hand, understand free-form gestures by synthesizing the gesture, context, experience, and common sense. In this way, the user does not need to learn, demonstrate, or associate gestures. We introduce GestureGPT, a free-form hand gesture understanding framework that mimics human gesture understanding procedures to enable a natural free-form gestural interface. Our framework leverages multiple Large Language Model agents to manage and synthesize gesture and context information, then infers the interaction intent by associating the gesture with an interface function. More specifically, our triple-agent framework includes a Gesture Description Agent that automatically segments and formulates natural language descriptions of hand poses and movements based on hand landmark coordinates. The description is deciphered by a Gesture Inference Agent through self-reasoning and querying about the interaction context (e.g., interaction history, gaze data), which is managed by a Context Management Agent. Following iterative exchanges, the Gesture Inference Agent discerns the user's intent by grounding it to an interactive function. We validated our framework offline under two real-world scenarios: smart home control and online video streaming. The average zero-shot Top-1/Top-5 grounding accuracies are 44.79%/83.59% for smart home tasks and 37.50%/73.44% for video streaming tasks. We also provide an extensive discussion that includes rationale for model selection, generalizability, and future research directions for a practical system etc.

cs.CL↗

arXiv2024

AUGlasses: Continuous Action Unit based Facial Reconstruction with Low-power IMUs on Smart Glasses

Recent advancements in augmented reality (AR) have enabled the use of various sensors on smart glasses for applications like facial reconstruction, which is vital to improve AR experiences for virtual social activities. However, the size and power constraints of smart glasses demand a miniature and low-power sensing solution. AUGlasses achieves unobtrusive low-power facial reconstruction by placing inertial measurement units (IMU) against the temporal area on the face to capture the skin deformations, which are caused by facial muscle movements. These IMU signals, along with historical data on facial action units (AUs), are processed by a transformer-based deep learning model to estimate AU intensities in real-time, which are then used for facial reconstruction. Our results show that AUGlasses accurately predicts the strength (0-5 scale) of 14 key AUs with a cross-user mean absolute error (MAE) of 0.187 (STD = 0.025) and achieves facial reconstruction with a cross-user MAE of 1.93 mm (STD = 0.353). We also integrated various preprocessing and training techniques to ensure robust performance for continuous sensing. Micro-benchmark tests indicate that our system consistently performs accurate continuous facial reconstruction with a fine-tuned cross-user model, achieving an AU MAE of 0.35.

cs.HC↗

arXiv2024

xTrimoGene: An Efficient and Scalable Representation Learner for Single-Cell RNA-Seq Data

Advances in high-throughput sequencing technology have led to significant progress in measuring gene expressions at the single-cell level. The amount of publicly available single-cell RNA-seq (scRNA-seq) data is already surpassing 50M records for humans with each record measuring 20,000 genes. This highlights the need for unsupervised representation learning to fully ingest these data, yet classical transformer architectures are prohibitive to train on such data in terms of both computation and memory. To address this challenge, we propose a novel asymmetric encoder-decoder transformer for scRNA-seq data, called xTrimoGene$^α$ (or xTrimoGene for short), which leverages the sparse characteristic of the data to scale up the pre-training. This scalable design of xTrimoGene reduces FLOPs by one to two orders of magnitude compared to classical transformers while maintaining high accuracy, enabling us to train the largest transformer models over the largest scRNA-seq dataset today. Our experiments also show that the performance of xTrimoGene improves as we scale up the model sizes, and it also leads to SOTA performance over various downstream tasks, such as cell type annotation, perturb-seq effect prediction, and drug combination prediction. xTrimoGene model is now available for use as a service via the following link: https://api.biomap.com/xTrimoGene/apply.

cs.LG↗

arXiv2024

Deep Submillimetre and Radio Observations in the SSA22 Field. II. Sub-millimetre source catalogue and number counts

We present the deepest 850 $μ$m map of the SSA22 field to date, utilizing a combination of new and archival observations taken with SCUBA-2, mounted at the James Clerk Maxwell Telescope (JCMT). The mapped area covers an effective region of approximately 0.34 deg$^2$, achieving a boundary sensitivity of 2 mJy beam$^{-1}$, with the deepest central coverage reaching a depth of $σ_\text{rms}$ $\sim$ 0.79 mJy beam$^{-1}$, the confusion noise is estimated to be $\sim$ 0.43 mJy beam$^{-1}$. A catalogue of 850 $μ$m sources in the SSA22 field is generated, identifying 390 sources with single-to-noise ratios above 3.5, out of which 92 sources exceed 5$σ$. The derived intrinsic number counts at 850 $μ$m are found to be in excellent agreement with published surveys. Interestingly, the SSA22 number counts also exhibit an upturn in the brighter flux region, likely attributed to local emitters or lensing objects within the field. On the scale of $\sim$ 0.3 deg$^2$, the 850 $μ$m number counts are unaffected by cosmic variance and align with the blank field. In the deep region ($σ_\text{rms}$ $\leqslant$ 1 mJy), the counts for fluxes below 8 mJy are consistent with the blank field, and the excess in the brighter regime is not significant. Due to the limited number of very bright sources and the insubstantial cosmic variance in our field, we attribute the fluctuations in the number counts primarily to Poisson noise. The SCUBA-2 850 $μ$m detection in the SSA22 field does not exhibit indications of overdensity.

astro-ph.GA↗

arXiv2023

Joint Beamforming for RIS Aided Full-Duplex Integrated Sensing and Uplink Communication

This paper studies integrated sensing and communication (ISAC) technology in a full-duplex (FD) uplink communication system. As opposed to the half-duplex system, where sensing is conducted in a first-emit-then-listen manner, FD ISAC system emits and listens simultaneously and hence conducts uninterrupted target sensing. Besides, impressed by the recently emerging reconfigurable intelligent surface (RIS) technology, we also employ RIS to improve the self-interference (SI) suppression and signal processing gain. As will be seen, the joint beamforming, RIS configuration and mobile users' power allocation is a difficult optimization problem. To resolve this challenge, via leveraging the cutting-the-edge majorization-minimization (MM) and penalty-dual-decomposition (PDD) methods, we develop an iterative solution that optimizes all variables via using convex optimization techniques. Numerical results demonstrate the effectiveness of our proposed solution and the great benefit of employing RIS in the FD ISAC system.

cs.IT↗

arXiv2023

Bridging the Granularity Gap for Acoustic Modeling

While Transformer has become the de-facto standard for speech, modeling upon the fine-grained frame-level features remains an open challenge of capturing long-distance dependencies and distributing the attention weights. We propose \textit{Progressive Down-Sampling} (PDS) which gradually compresses the acoustic features into coarser-grained units containing more complete semantic information, like text-level representation. In addition, we develop a representation fusion method to alleviate information loss that occurs inevitably during high compression. In this way, we compress the acoustic features into 1/32 of the initial length while achieving better or comparable performances on the speech recognition task. And as a bonus, it yields inference speedups ranging from 1.20$\times$ to 1.47$\times$. By reducing the modeling burden, we also achieve competitive results when training on the more challenging speech translation task.

cs.CL↗

arXiv2022

Robust Secure Transmission Design for IRS-Assisted mmWave Cognitive Radio Networks

Cognitive radio networks (CRNs) and millimeter wave (mmWave) communications are two major technologies to enhance the spectrum efficiency (SE). Considering that the SE improvement in the CRNs is limited due to the interference temperature imposed on the primary user (PU), and the severe path loss and high directivity in mmWave communications make it vulnerable to blockage events, we introduce an intelligent reflecting surface (IRS) into mmWave CRNs. Due to the estimation mismatch and the passivity of Eavesdroppers (Eves), perfect channel state information (CSI) of wiretap links is challenging to obtain, which promotes our research on robust secure beamforming (BF) design in the IRS-assisted mmWave CRNs. This paper considers the collaborate scenario of Eves, which allows us to investigate the BF design in the harsh eavesdropping environment. Specifically, by using a uniform linear array (ULA) at the cognitive base station (CBS) and a uniform planar array (UPA) at the IRS, and supposing that imperfect CSIs of angle-of-departures for wiretap links are known, we formulate a constrained problem to maximize the worst-case achievable secrecy rate (ASR) of the secondary user (SU) by jointly designing the transmit BF at the CBS and reflect BF at the IRS. To solve the non-convex problem with coupled variables, an efficient alternating optimization algorithm is proposed. Finally, simulation results indicate that the ASR performance of our proposed algorithm has a small gap with that of the optimal solution with perfect CSI compared with the other benchmarks.

cs.IT↗

arXiv2022

ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation

Residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODE). This paper explores a deeper relationship between Transformer and numerical ODE methods. We first show that a residual block of layers in Transformer can be described as a higher-order solution to ODE. Inspired by this, we design a new architecture, {\it ODE Transformer}, which is analogous to the Runge-Kutta method that is well motivated in ODE. As a natural extension to Transformer, ODE Transformer is easy to implement and efficient to use. Experimental results on the large-scale machine translation, abstractive summarization, and grammar error correction tasks demonstrate the high genericity of ODE Transformer. It can gain large improvements in model performance over strong baselines (e.g., 30.77 and 44.11 BLEU scores on the WMT'14 English-German and English-French benchmarks) at a slight cost in inference efficiency.

cs.CL↗

arXiv2021

Secure and Energy Efficient Transmission for IRS-Assisted Cognitive Radio Networks

The spectrum efficiency (SE) and security of the secondary users (SUs) in the cognitive radio networks (CRNs) have become two main issues due to the limitation interference to the primary users (PUs) and the shared spectrum with the PUs. Intelligent reflecting surface (IRS) has been recently proposed as a revolutionary technique which can help to enhance the SE and physical layer security of wireless communications. This paper investigates the application of IRS in an underlay CRN, where a multi-antenna cognitive base station (CBS) utilizes spectrum assigned to the PU to communicate with a SU via IRS in the presence of multiple coordinated eavesdroppers (Eves). To achieve the trade-off between the secrecy rate (SR) and energy consumption, we investigate the secrecy energy efficiency (SEE) maximization problem by jointly designing the transmit beamforming at the CBS and the reflect beamforming at the IRS. To solve the non-convex problem with coupled variables, we propose an iterative alternating optimization algorithm to solve the sub-problems alternately, by utilizing an iterative penalty function based algorithm for sub-problem 1 and the difference of two-convex functions method for sub-problem 2. Furthermore, we provide a second-order-cone-programming (SOCP) approximation approach to reduce the computational complexity. Finally, the simulation results demonstrate that IRS can help significantly improve the SE and enhance the physical layer security in the CRNs. Moreover, the effectiveness and superiority of our proposed algorithm in achieving the trade-off between the SR and energy consumption are verified.

cs.IT↗

arXiv2021

Spectral compression by phase doubling in second harmonic generation

In second harmonic generation, the phase of the optical field is doubled which has important implication. Here the phase doubling effect is utilized to solve a long-standing challenge in power scaling of single frequency laser. When a (-π/2, π/2) binary phase modulation is applied to a single frequency seed laser to broaden the spectrum and suppress the stimulated Brillouin scattering in high power fiber amplifier, the second harmonic of the phase-modulated laser will return to single frequency, because the (-π/2, π/2) modulation is doubled to (-π, π) for the second harmonic. A compression rate as high as 95% is demonstrated in the experiment limited by the electronic bandwidth of the setup, which can be improved with optimized devices.

physics.optics↗

arXiv2021

ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation

It has been found that residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODEs). In this paper, we explore a deeper relationship between Transformer and numerical methods of ODEs. We show that a residual block of layers in Transformer can be described as a higher-order solution to ODEs. This leads us to design a new architecture (call it ODE Transformer) analogous to the Runge-Kutta method that is well motivated in ODEs. As a natural extension to Transformer, ODE Transformer is easy to implement and parameter efficient. Our experiments on three WMT tasks demonstrate the genericity of this model, and large improvements in performance over several strong baselines. It achieves 30.76 and 44.11 BLEU scores on the WMT'14 En-De and En-Fr test data. This sets a new state-of-the-art on the WMT'14 En-Fr task.

cs.CL↗

arXiv2021

IFoodCloud: A Platform for Real-time Sentiment Analysis of Public Opinion about Food Safety in China

The Internet contains a wealth of public opinion on food safety, including views on food adulteration, food-borne diseases, agricultural pollution, irregular food distribution, and food production issues. In order to systematically collect and analyse public opinion on food safety, we developed IFoodCloud, a platform for the real-time sentiment analysis of public opinion on food safety in China. It collects data from more than 3,100 public sources that can be used to explore public opinion trends, public sentiment, and regional attention differences of food safety incidents. At the same time, we constructed a sentiment classification model using multiple lexicon-based and deep learning-based algorithms integrated with IFoodCloud that provide an unprecedented rapid means of understanding the public sentiment toward specific food safety incidents. Our best model's F1-score achieved 0.9737. Further, three real-world cases are presented to demonstrate the application and robustness. IFoodCloud could be considered a valuable tool for promote scientisation of food safety supervision and risk communication.

cs.CL↗

arXiv2018

Deep learning-based super-resolution in coherent imaging systems

We present a deep learning framework based on a generative adversarial network (GAN) to perform super-resolution in coherent imaging systems. We demonstrate that this framework can enhance the resolution of both pixel size-limited and diffraction-limited coherent imaging systems. We experimentally validated the capabilities of this deep learning-based coherent imaging approach by super-resolving complex images acquired using a lensfree on-chip holographic microscope, the resolution of which was pixel size-limited. Using the same GAN-based approach, we also improved the resolution of a lens-based holographic imaging system that was limited in resolution by the numerical aperture of its objective lens. This deep learning-based super-resolution framework can be broadly applied to enhance the space-bandwidth product of coherent imaging systems using image data and convolutional neural networks, and provides a rapid, non-iterative method for solving inverse image reconstruction or enhancement problems in optics.

cs.CV↗

arXiv2013

On quasi-periodic solutions of the discrete Chen-Lee-Liu hierarchy

Resorting to the Lax matrix and elliptic variables, the discrete Chen-Lee-Liu hierarchy is decomposed into solvable ordinary differential equations. Based on the theory of algebraic curve, the continuous flow and discrete flow related to the discrete Chen-Lee-Liu hierarchy are straightened under the Abel-Jacobi coordinates. The meromorphic function $ϕ$, the Baker-Akhiezer vector $\barψ$ and the hyperelliptic curve $\mathcal{K}_N$ are introduced, by which quasi-periodic solutions of the discrete Chen-Lee-Liu hierarchy are constructed according to the asymptotic properties and the algebro-geometric characters of $ϕ,\ \barψ$ and $\mathcal{K}_N$.

math.AG↗