arXiv subjects

Xinyu Gao

Publications and source records attributed to Xinyu Gao.

At least 55 records · Page 3Linked to original sources

arXiv2023

Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction

Implicit neural representation has paved the way for new approaches to dynamic scene reconstruction and rendering. Nonetheless, cutting-edge dynamic neural rendering methods rely heavily on these implicit representations, which frequently struggle to capture the intricate details of objects in the scene. Furthermore, implicit methods have difficulty achieving real-time rendering in general dynamic scenes, limiting their use in a variety of tasks. To address the issues, we propose a deformable 3D Gaussians Splatting method that reconstructs scenes using 3D Gaussians and learns them in canonical space with a deformation field to model monocular dynamic scenes. We also introduce an annealing smoothing training mechanism with no extra overhead, which can mitigate the impact of inaccurate poses on the smoothness of time interpolation tasks in real-world datasets. Through a differential Gaussian rasterizer, the deformable 3D Gaussians not only achieve higher rendering quality but also real-time rendering speed. Experiments show that our method outperforms existing methods significantly in terms of both rendering quality and speed, making it well-suited for tasks such as novel-view synthesis, time interpolation, and real-time rendering.

cs.CV↗

arXiv2023

SIRe-IR: Inverse Rendering for BRDF Reconstruction with Shadow and Illumination Removal in High-Illuminance Scenes

Implicit neural representation has opened up new possibilities for inverse rendering. However, existing implicit neural inverse rendering methods struggle to handle strongly illuminated scenes with significant shadows and indirect illumination. The existence of shadows and reflections can lead to an inaccurate understanding of scene geometry, making precise factorization difficult. To this end, we present SIRe-IR, an implicit neural inverse rendering approach that uses non-linear mapping and regularized visibility estimation to decompose the scene into environment map, albedo, and roughness. By accurately modeling the indirect radiance field, normal, visibility, and direct light simultaneously, we are able to remove both shadows and indirect illumination in materials without imposing strict constraints on the scene. Even in the presence of intense illumination, our method recovers high-quality albedo and roughness with no shadow interference. SIRe-IR outperforms existing methods in both quantitative and qualitative evaluations.

cs.CV↗

arXiv2023

Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning

Recent advancements enlarge the capabilities of large language models (LLMs) in zero-shot image-to-text generation and understanding by integrating multi-modal inputs. However, such success is typically limited to English scenarios due to the lack of large-scale and high-quality non-English multi-modal resources, making it extremely difficult to establish competitive counterparts in other languages. In this paper, we introduce the Ziya-Visual series, a set of bilingual large-scale vision-language models (LVLMs) designed to incorporate visual semantics into LLM for multi-modal dialogue. Composed of Ziya-Visual-Base and Ziya-Visual-Chat, our models adopt the Querying Transformer from BLIP-2, further exploring the assistance of optimization schemes such as instruction tuning, multi-stage training and low-rank adaptation module for visual-language alignment. In addition, we stimulate the understanding ability of GPT-4 in multi-modal scenarios, translating our gathered English image-text datasets into Chinese and generating instruction-response through the in-context learning method. The experiment results demonstrate that compared to the existing LVLMs, Ziya-Visual achieves competitive performance across a wide range of English-only tasks including zero-shot image-text retrieval, image captioning, and visual question answering. The evaluation leaderboard accessed by GPT-4 also indicates that our models possess satisfactory image-text understanding and generation capabilities in Chinese multi-modal scenario dialogues. Code, demo and models are available at ~\url{https://huggingface.co/IDEA-CCNL/Ziya-BLIP2-14B-Visual-v1}.

cs.CL↗

arXiv2023

FEcMD: A multi-physics and multi-scale computational program for electron emission characteristics dynamically coupled with atomic structure in metal nano-emitters

Field emission coupled with molecular dynamics simulation (FEcMD) software package is a computational tool for studying the electron emission characteristics and the atomic structure evolution of micro- and nano-protrusions made of pure metals or multi-component alloys by means of multi-physics and multi-scale methodology. The implementations of molecular dynamics, the electrodynamics, and the heat conduction in FEcMD program are addressed. For molecular dynamics simulation, the Lennard-Jones potentials, embedded atomic method (EAM), and moment tensor potentials (MTP) are fully supported for both alloys and pure metals. In the electrodynamics, the FEcMD program incorporates the space charge fields (space charge potential and exchange-correlation effects) in the Wentzel-Kramers-Brillouin-Jeffreys (WKBJ) approximation to evaluate the field emission current density more reliably for nano-gaps between two metal electrodes. Additionally, the advanced two-temperature heat conduction model is implemented in FEcMD program, and which provides more reliable descriptions for the temperature evolutions of electron and phonon subsystems under the radiofrequency (RF) or pulse electric fields. Comprehensive benchmark tests are performed for each module in FEcMD software to validate the numerical results, and also to access the accuracy and efficiency of the implemented algorithms. Finally, some typical applications of FEcMD program are also demonstrated for understanding the evolution of temperature and electric field coupled with the dynamic changing of atomic structures for metal micro- and nano-protrusions.

physics.comp-ph↗

arXiv2023

Benchmarking Robustness of AI-Enabled Multi-sensor Fusion Systems: Challenges and Opportunities

Multi-Sensor Fusion (MSF) based perception systems have been the foundation in supporting many industrial applications and domains, such as self-driving cars, robotic arms, and unmanned aerial vehicles. Over the past few years, the fast progress in data-driven artificial intelligence (AI) has brought a fast-increasing trend to empower MSF systems by deep learning techniques to further improve performance, especially on intelligent systems and their perception systems. Although quite a few AI-enabled MSF perception systems and techniques have been proposed, up to the present, limited benchmarks that focus on MSF perception are publicly available. Given that many intelligent systems such as self-driving cars are operated in safety-critical contexts where perception systems play an important role, there comes an urgent need for a more in-depth understanding of the performance and reliability of these MSF systems. To bridge this gap, we initiate an early step in this direction and construct a public benchmark of AI-enabled MSF-based perception systems including three commonly adopted tasks (i.e., object detection, object tracking, and depth completion). Based on this, to comprehensively understand MSF systems' robustness and reliability, we design 14 common and realistic corruption patterns to synthesize large-scale corrupted datasets. We further perform a systematic evaluation of these systems through our large-scale evaluation. Our results reveal the vulnerability of the current AI-enabled MSF perception systems, calling for researchers and practitioners to take robustness and reliability into account when designing AI-enabled MSF.

cs.SE↗

arXiv2023

A Camera free fiber speckle wavemeter

Recovering the wavelength from disordered speckle patterns has become an exciting prospect as a wavelength measurement method due to its high resolution and simple design. In previous studies, panel cameras have been used to detect the subtle differences between speckle patterns. However, the volume, bandwidth, sensitivity, and cost (in non-visible bands) associated with panel cameras have hindered their utility in broader applications, especially in high speed and low-cost measurements. In this work, we broke the limitations imposed by panel cameras by using a quadrant detector (QD) to capture the speckle images. In the scheme of QD detection, speckle images are directly filtered by convolution, where the kernel is equal to one quarter of a speckle pattern. First, we proposed an up-sampling algorithm to pre-process the QD data. Then a new convolution neural network (CNN) based algorithm, shallow residual network (SRN), was proposed to train the up-sampled images. The experimental results show that a resolution of 4 fm (~ 0.5 MHz) was achieved at 1550nm with an updating speed of ~ 1 kHz. More importantly, the SRN shows excellent robustness. The wavelength can be precisely reconstructed from raw QD data without any averaging, even where there exists apparent noise. The low-cost, simple structure, high speed and robustness of this design promote the speckle-based wavemeter to the industrial grade. In addition, without the restriction of panel cameras, it is believed that this wavemeter opens new routes in many other fields, such as distributed optical fiber sensors, optical communications, and laser frequency stabilization.

physics.ins-det↗

arXiv2023

Multi-Objective Optimisation of URLLC-Based Metaverse Services

Metaverse aims for building a fully immersive virtual shared space, where the users are able to engage in various activities. To successfully deploy the service for each user, the Metaverse service provider and network service provider generally localise the user first and then support the communication between the base station (BS) and the user. A reconfigurable intelligent surface (RIS) is capable of creating a reflected link between the BS and the user to enhance line-of-sight. Furthermore, the new key performance indicators (KPIs) in Metaverse, such as its energy-consumption-dependent total service cost and transmission latency, are often overlooked in ultra-reliable low latency communication (URLLC) designs, which have to be carefully considered in next-generation URLLC (xURLLC) regimes. In this paper, our design objective is to jointly optimise the transmit power, the RIS phase shifts, and the decoding error probability to simultaneously minimise the total service cost and transmission latency and approach the Pareto Front (PF). We conceive a twin-stage central controller, which aims for localising the users first and then supports the communication between the BS and users. In the first stage, we localise the Metaverse users, where the stochastic gradient descent (SGD) algorithm is invoked for accurate user localisation. In the second stage, a meta-learning-based position-dependent multi-objective soft actor and critic (MO-SAC) algorithm is proposed to approach the PF between the total service cost and transmission latency and to further optimise the latency-dependent reliability. Our numerical results demonstrate that ...

cs.IT↗

arXiv2023

DRL Enabled Coverage and Capacity Optimization in STAR-RIS Assisted Networks

Simultaneously transmitting and reflecting reconfigurable intelligent surfaces (STAR-RISs) is a promising passive device that contributes to a full-space coverage via transmitting and reflecting the incident signal simultaneously. As a new paradigm in wireless communications, how to analyze the coverage and capacity performance of STAR-RISs becomes essential but challenging. To solve the coverage and capacity optimization (CCO) problem in STAR-RIS assisted networks, a multi-objective proximal policy optimization (MO-PPO) algorithm is proposed to handle long-term benefits than conventional optimization algorithms. To strike a balance between each objective, the MO-PPO algorithm provides a set of optimal solutions to form a Pareto front (PF), where any solution on the PF is regarded as an optimal result. Moreover, in order to improve the performance of the MO-PPO algorithm, two update strategies, i.e., action-value-based update strategy (AVUS) and loss function-based update strategy (LFUS), are investigated. For the AVUS, the improved point is to integrate the action values of both coverage and capacity and then update the loss function. For the LFUS, the improved point is only to assign dynamic weights for both loss functions of coverage and capacity, while the weights are calculated by a min-norm solver at every update. The numerical results demonstrated that the investigated update strategies outperform the fixed weights MO optimization algorithms in different cases, which includes a different number of sample grids, the number of STAR-RISs, the number of elements in the STAR-RISs, and the size of STAR-RISs. Additionally, the STAR-RIS assisted networks achieve better performance than conventional wireless networks without STAR-RISs. Moreover, with the same bandwidth, millimeter wave is able to provide higher capacity than sub-6 GHz, but at a cost of smaller coverage.

cs.IT↗

arXiv2023

Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence

Nowadays, foundation models become one of fundamental infrastructures in artificial intelligence, paving ways to the general intelligence. However, the reality presents two urgent challenges: existing foundation models are dominated by the English-language community; users are often given limited resources and thus cannot always use foundation models. To support the development of the Chinese-language community, we introduce an open-source project, called Fengshenbang, which leads by the research center for Cognitive Computing and Natural Language (CCNL). Our project has comprehensive capabilities, including large pre-trained models, user-friendly APIs, benchmarks, datasets, and others. We wrap all these in three sub-projects: the Fengshenbang Model, the Fengshen Framework, and the Fengshen Benchmark. An open-source roadmap, Fengshenbang, aims to re-evaluate the open-source community of Chinese pre-trained large-scale models, prompting the development of the entire Chinese large-scale model community. We also want to build a user-centered open-source ecosystem to allow individuals to access the desired models to match their computing resources. Furthermore, we invite companies, colleges, and research institutions to collaborate with us to build the large-scale open-source model-based ecosystem. We hope that this project will be the foundation of Chinese cognitive intelligence.

cs.CL↗

arXiv2023

Intelligent Trajectory Design for RIS-NOMA aided Multi-robot Communications

A novel reconfigurable intelligent surface-aided multi-robot network is proposed, where multiple mobile robots are served by an access point (AP) through non-orthogonal multiple access (NOMA). The goal is to maximize the sum-rate of whole trajectories for the multi-robot system by jointly optimizing trajectories and NOMA decoding orders of robots, phase-shift coefficients of the RIS, and the power allocation of the AP, subject to predicted initial and final positions of robots and the quality of service (QoS) of each robot. To tackle this problem, an integrated machine learning (ML) scheme is proposed, which combines long short-term memory (LSTM)-autoregressive integrated moving average (ARIMA) model and dueling double deep Q-network (D$^{3}$QN) algorithm. For initial and final position prediction for robots, the LSTM-ARIMA is able to overcome the problem of gradient vanishment of non-stationary and non-linear sequences of data. For jointly determining the phase shift matrix and robots' trajectories, D$^{3}$QN is invoked for solving the problem of action value overestimation. Based on the proposed scheme, each robot holds an optimal trajectory based on the maximum sum-rate of a whole trajectory, which reveals that robots pursue long-term benefits for whole trajectory design. Numerical results demonstrated that: 1) LSTM-ARIMA model provides high accuracy predicting model; 2) The proposed D$^{3}$QN algorithm can achieve fast average convergence; and 3) RIS-NOMA networks have superior network performance compared to RIS-aided orthogonal counterparts.

cs.RO↗

arXiv2023

Reconfigurable Massive MIMO: Harnessing the Power of the Electromagnetic Domain for Enhanced Information Transfer

The capacity of commercial massive multiple-input multiple-output (mMIMO) systems is constrained by the limited array aperture at the base station, and cannot meet the ever-increasing traffic demands of wireless networks. Given the array aperture, holographic MIMO with infinitesimal antenna spacing can maximize the capacity, but is physically unrealizable. As a promising alternative, reconfigurable mMIMO is proposed to harness the unexploited power of the electromagnetic (EM) domain for enhanced information transfer. Specifically, the reconfigurable pixel antenna technology provides each antenna with an adjustable EM radiation (EMR) pattern, introducing extra degrees of freedom for information transfer in the EM domain. In this article, we present the concept and benefits of availing the EMR domain for mMIMO transmission. Moreover, we propose a viable architecture for reconfigurable mMIMO systems, and the associated system model and downlink precoding are also discussed. In particular, a three-level precoding scheme is proposed, and simulation results verify its considerable spectral and energy efficiency advantages compared to traditional mMIMO systems. Finally, we further discuss the challenges, insights, and prospects of deploying reconfigurable mMIMO, along with the associated hardware, algorithms, and fundamental theory.

cs.IT↗

arXiv2022

TCBERT: A Technical Report for Chinese Topic Classification BERT

Bidirectional Encoder Representations from Transformers or BERT~\cite{devlin-etal-2019-bert} has been one of the base models for various NLP tasks due to its remarkable performance. Variants customized for different languages and tasks are proposed to further improve the performance. In this work, we investigate supervised continued pre-training~\cite{gururangan-etal-2020-dont} on BERT for Chinese topic classification task. Specifically, we incorporate prompt-based learning and contrastive learning into the pre-training. To adapt to the task of Chinese topic classification, we collect around 2.1M Chinese data spanning various topics. The pre-trained Chinese Topic Classification BERTs (TCBERTs) with different parameter sizes are open-sourced at \url{https://huggingface.co/IDEA-CCNL}.

cs.CL↗

arXiv2022

Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective

We propose a new paradigm for zero-shot learners that is format agnostic, i.e., it is compatible with any format and applicable to a list of language tasks, such as text classification, commonsense reasoning, coreference resolution, and sentiment analysis. Zero-shot learning aims to train a model on a given task such that it can address new learning tasks without any additional training. Our approach converts zero-shot learning into multiple-choice tasks, avoiding problems in commonly used large-scale generative models such as FLAN. It not only adds generalization ability to models but also significantly reduces the number of parameters. Our method shares the merits of efficient training and deployment. Our approach shows state-of-the-art performance on several benchmarks and produces satisfactory results on tasks such as natural language inference and text classification. Our model achieves this success with only 235M parameters, which is substantially smaller than state-of-the-art models with billions of parameters. The code and pre-trained models are available at https://github.com/IDEA-CCNL/Fengshenbang-LM .

cs.CL↗

arXiv2022

Coverage and Capacity Optimization in STAR-RISs Assisted Networks: A Machine Learning Approach

Coverage and capacity are the important metrics for performance evaluation in wireless networks, while the coverage and capacity have several conflicting relationships, e.g. high transmit power contributes to large coverage but high inter-cell interference reduces the capacity performance. Therefore, in order to strike a balance between the coverage and capacity, a novel model is proposed for the coverage and capacity optimization of simultaneously transmitting and reflecting reconfigurable intelligent surfaces (STAR-RISs) assisted networks. To solve the coverage and capacity optimization (CCO) problem, a machine learning-based multi-objective optimization algorithm, i.e., the multi-objective proximal policy optimization (MO-PPO) algorithm, is proposed. In this algorithm, a loss function-based update strategy is the core point, which is able to calculate weights for both loss functions of coverage and capacity by a min-norm solver at each update. The numerical results demonstrate that the investigated update strategy outperforms the fixed weight-based MO algorithms.

cs.IT↗

arXiv2021

Robotic Communications for 5G and Beyond: Challenges and Research Opportunities

The ongoing surge in applications of robotics brings both opportunities and challenges for the fifth-generation (5G) and beyond (B5G) of communication networks. This article focuses on 5G/B5G-enabled terrestrial robotic communications with an emphasis on distinct characteristics of such communications. Firstly, signal and spatial modeling for robotic communications are presented. To elaborate further, both the benefits and challenges derived from robots' mobility are discussed. As a further advance, a novel simultaneous localization and radio mapping (SLARM) framework is proposed for integrating localization and communications into robotic networks. Furthermore, dynamic trajectory design and resource allocation for both indoor and outdoor robots are provided to verify the performance of robotic communications in the context of typical robotic application scenarios.

cs.RO↗

arXiv2021

Human Activity and Mobility Data Reveal Disparities in Exposure Risk Reduction Indicators among Socially Vulnerable Populations during COVID-19

Non-pharmacologic interventions (NPIs) are one method to mitigate the spread and effects of the COVID-19 pandemic in the United States. NPIs promote protective actions to reduce exposure risk and can reduce mobility patterns within communities. Growing research literature suggests that socially vulnerable populations are disproportionately impacted with higher infection and higher fatality rates of COVID-19, though there is limited understanding of the underlying mechanisms to this health disparity. Thus, the research examines two distinct and complimentary datasets at a granular scale for five urban locations. Through statistical and spatial analyses, the research extensively investigates the exposure risk reduction of socially vulnerable populations due to NPIs. The mobility dataset tracks population movement across ZIP codes; it is used for an origin-destination network analysis. The population activity dataset is based on the number of visits from census block groups (CBG) to points of interest (POIs), such as grocery stores, restaurants, education centers, and medical facilities; it is used for network analysis of population-facilities interactions. The mobility dataset showed that, after the implementation of NPIs, socially vulnerable populations engaged in increased mobility in the form of inflow between ZIP code areas. Similarly, population activity analysis showed an increased exposure risk for socially vulnerable populations based on a greater number of inflow visits of CBGs to POIs, which increases the risk of contact at POIs, and a greater number of outflow visits from POIs to home CBGs, which increases risk of transmission within CBGs. These findings can assist emergency planners and public health officials in comprehending how different groups are able to implement protective actions and can inform more equitable and data-driven NPI policies for future epidemics.

physics.soc-ph↗

arXiv2021

Improvement of image classification by multiple optical scattering

Multiple optical scattering occurs when light propagates in a non-uniform medium. During the multiple scattering, images were distorted and the spatial information they carried became scrambled. However, the image information is not lost but presents in the form of speckle patterns (SPs). In this study, we built up an optical random scattering system based on an LCD and an RGB laser source. We found that the image classification can be improved by the help of random scattering which is considered as a feedforward neural network to extracts features from image. Along with the ridge classification deployed on computer, we achieved excellent classification accuracy higher than 94%, for a variety of data sets covering medical, agricultural, environmental protection and other fields. In addition, the proposed optical scattering system has the advantages of high speed, low power consumption, and miniaturization, which is suitable for deploying in edge computing applications.

cs.CV↗

arXiv2021

Deep-learning-based Hyperspectral imaging through a RGB camera

Hyperspectral image (HSI) contains both spatial pattern and spectral information which has been widely used in food safety, remote sensing, and medical detection. However, the acquisition of hyperspectral images is usually costly due to the complicated apparatus for the acquisition of optical spectrum. Recently, it has been reported that HSI can be reconstructed from single RGB image using convolution neural network (CNN) algorithms. Compared with the traditional hyperspectral cameras, the method based on CNN algorithms is simple, portable and low cost. In this study, we focused on the influence of the RGB camera spectral sensitivity (CSS) on the HSI. A Xenon lamp incorporated with a monochromator were used as the standard light source to calibrate the CSS. And the experimental results show that the CSS plays a significant role in the reconstruction accuracy of an HSI. In addition, we proposed a new HSI reconstruction network where the dimensional structure of the original hyperspectral datacube was modified by 3D matrix transpose to improve the reconstruction accuracy.

eess.IV↗