arXiv subjects

Jie Zhang

Publications and source records attributed to Jie Zhang.

At least 559 records · Page 31Linked to original sources

arXiv2024

MaTe3D: Mask-guided Text-based 3D-aware Portrait Editing

3D-aware portrait editing has a wide range of applications in multiple fields. However, current approaches are limited due that they can only perform mask-guided or text-based editing. Even by fusing the two procedures into a model, the editing quality and stability cannot be ensured. To address this limitation, we propose \textbf{MaTe3D}: mask-guided text-based 3D-aware portrait editing. In this framework, first, we introduce a new SDF-based 3D generator which learns local and global representations with proposed SDF and density consistency losses. This enhances masked-based editing in local areas; second, we present a novel distillation strategy: Conditional Distillation on Geometry and Texture (CDGT). Compared to exiting distillation strategies, it mitigates visual ambiguity and avoids mismatch between texture and geometry, thereby producing stable texture and convincing geometry while editing. Additionally, we create the CatMask-HQ dataset, a large-scale high-resolution cat face annotation for exploration of model generalization and expansion. We perform expensive experiments on both the FFHQ and CatMask-HQ datasets to demonstrate the editing quality and stability of the proposed method. Our method faithfully generates a 3D-aware edited face image based on a modified mask and a text prompt. Our code and models will be publicly released.

cs.CV↗

arXiv2024

The USTC-NERCSLIP Systems for The ICMC-ASR Challenge

This report describes the submitted system to the In-Car Multi-Channel Automatic Speech Recognition (ICMC-ASR) challenge, which considers the ASR task with multi-speaker overlapping and Mandarin accent dynamics in the ICMC case. We implement the front-end speaker diarization using the self-supervised learning representation based multi-speaker embedding and beamforming using the speaker position, respectively. For ASR, we employ an iterative pseudo-label generation method based on fusion model to obtain text labels of unsupervised data. To mitigate the impact of accent, an Accent-ASR framework is proposed, which captures pronunciation-related accent features at a fine-grained level and linguistic information at a coarse-grained level. On the ICMC-ASR eval set, the proposed system achieves a CER of 13.16% on track 1 and a cpCER of 21.48% on track 2, which significantly outperforms the official baseline system and obtains the first rank on both tracks.

eess.AS↗

arXiv2024

Complementary Fusion of Deep Network and Tree Model for ETA Prediction

Estimated time of arrival (ETA) is a very important factor in the transportation system. It has attracted increasing attentions and has been widely used as a basic service in navigation systems and intelligent transportation systems. In this paper, we propose a novel solution to the ETA estimation problem, which is an ensemble on tree models and neural networks. We proved the accuracy and robustness of the solution on the A/B list and finally won first place in the SIGSPATIAL 2021 GISCUP competition.

cs.LG↗

arXiv2024

Decision Transformer for IRS-Assisted Systems with Diffusion-Driven Generative Channels

In this paper, we propose a novel diffusion-decision transformer (D2T) architecture to optimize the beamforming strategies for intelligent reflecting surface (IRS)-assisted multiple-input single-output (MISO) communication systems. The first challenge lies in the expensive computation cost to recover the real-time channel state information (CSI) from the received pilot signals, which usually requires prior knowledge of the channel distributions. To reduce the channel estimation complexity, we adopt a diffusion model to automatically learn the mapping between the received pilot signals and channel matrices in a model-free manner. The second challenge is that, the traditional optimization or reinforcement learning (RL) algorithms cannot guarantee the optimality of the beamforming policies once the channel distribution changes, and it is costly to resolve the optimized strategies. To enhance the generality of the decision models over varying channel distributions, we propose an offline pre-training and online fine-tuning decision transformer (DT) framework, wherein we first pre-train the DT offline with the data samples collected by the RL algorithms under diverse channel distributions, and then fine-tune the DT online with few-shot samples under a new channel distribution for a generalization purpose. Simulation results demonstrate that, compared with retraining RL algorithms, the proposed D2T algorithm boosts the convergence speed by 3 times with only a few samples from the new channel distribution while enhancing the average user data rate by 6%.

eess.SP↗

arXiv2024

Sparsity-Driven EEG Channel Selection for Brain-Assisted Speech Enhancement

Speech enhancement is widely used as a front-end to improve the speech quality in many audio systems, while it is hard to extract the target speech in multi-talker conditions without prior information on the speaker identity. It was shown that the auditory attention on the target speaker can be decoded from the electroencephalogram (EEG) of the listener implicitly. In this work, we therefore propose a novel end-to-end brain-assisted speech enhancement network (BASEN), which incorporates the listeners' EEG signals and adopts a temporal convolutional network together with a convolutional multi-layer cross attention module to fuse EEG-audio features. Considering that an EEG cap with sparse channels exhibits multiple benefits and in practice many electrodes might contribute marginally, we further propose two channel selection methods, called residual Gumbel selection and convolutional regularization selection. They are dedicated to tackling training instability and duplicated channel selections, respectively. Experimental results on a public dataset show the superiority of the proposed BASEN over existing approaches. The proposed channel selection methods can significantly reduce the amount of informative EEG channels with a negligible impact on the performance.

eess.AS↗

arXiv2024

ID-Animator: Zero-Shot Identity-Preserving Human Video Generation

Generating high-fidelity human video with specified identities has attracted significant attention in the content generation community. However, existing techniques struggle to strike a balance between training efficiency and identity preservation, either requiring tedious case-by-case fine-tuning or usually missing identity details in the video generation process. In this study, we present \textbf{ID-Animator}, a zero-shot human-video generation approach that can perform personalized video generation given a single reference facial image without further training. ID-Animator inherits existing diffusion-based video generation backbones with a face adapter to encode the ID-relevant embeddings from learnable facial latent queries. To facilitate the extraction of identity information in video generation, we introduce an ID-oriented dataset construction pipeline that incorporates unified human attributes and action captioning techniques from a constructed facial image pool. Based on this pipeline, a random reference training strategy is further devised to precisely capture the ID-relevant embeddings with an ID-preserving loss, thus improving the fidelity and generalization capacity of our model for ID-specific video generation. Extensive experiments demonstrate the superiority of ID-Animator to generate personalized human videos over previous models. Moreover, our method is highly compatible with popular pre-trained T2V models like animatediff and various community backbone models, showing high extendability in real-world applications for video generation where identity preservation is highly desired. Our codes and checkpoints are released at https://github.com/ID-Animator/ID-Animator.

cs.CV↗

arXiv2024

Modeling of axion and electromagnetic fields coupling in a particle-in-cell code

Axions have aroused widespread research interest because they can solve the strong CP problem and serve as a possible candidate for dark matter. Currently, people have explored a lot of axion detection experiments, including passively detecting the existing axions in the universe, and actively generating axions in the laboratory. Recently, axion-coupled laser-plasma interactions have been discussed as a novel method to detect axions. Petawatt (PW) lasers are considered as a powerful tool to study not only the vacuum polarization but also the axion coupling, due to their extreme fields. However, particle-in-cell (PIC) simulation is still missed in current studies, which limits the understanding of axion-coupled laser-plasma interactions. In this paper, we proposed the method to include the axion field and the coupling with electromagnetic (EM) fields in PIC codes. The axion wave equation and modified Maxwell's equations are numerically solved, while the EM field modulation from axions is considered as a first-order perturbation. Meanwhile, different axion field boundary conditions are considered to satisfy different simulation scenarios. The processes of conversions between axions and photons, and weak laser pulse propagation with axion effects are checked as benchmarks of the code. Such an extended PIC code may help researchers develop novel axion detection schemes based on laser-plasma interactions and provide a better understanding of axion-coupled astrophysical processes.

hep-ph↗

arXiv2024

Realized Stable BP-N at Ambient Pressure by Phosphorus Doping

Black phosphorus nitrogen (BP-N) is an attractive high-energy-density material. However, high-pressure synthesized BP-N will decompose at low-pressure and cannot be quenched to ambient conditions. Finding a method to stabilize it at 0 GPa is of great significance for its practical applications. However, unlike cg-N, LP-N, and HLP-N, it is always a metastable phase at high-pressure up to 260 GPa, and decomposes into chains at 23 GPa. Here, based on the first-principles simulations, we find that P atom doping can effectively reduce the synthesis pressure of BP-N and maintain its stability at 0 GPa. Uniform distribution of P atom dopants within the layer helps maintain the structural stability of BP-N layer at 0 GPa, while interlayer electrostatic interaction induced by N-P dipoles enhances its dynamic stability by eliminating interlayer slipping. Furthermore, pressure is conducive to enhancing the stability of BP-N and its doped forms by suppressing N-chain dissociation. For the configuration with 12.5% doping concentration, a gravimetric energy density of 8.07 kJ/g can be realized, which is nearly two times higher than TNT.

cond-mat.mtrl-sci↗

arXiv2024

From Optimal Transport to Efficient Mechanisms for the $m$-Capacitated Facilities Location Problem with Bayesian Agent

In this paper, we study of the $m$-Capacitated Facility Location Problem ($m$-CFLP) on the line from a Bayesian Mechanism Design perspective and propose a novel class of mechanisms: the \textit{Extended Ranking Mechanisms} (ERMs). We first show that an ERM is truthful if and only if it satisfies a system of inequalities that depends on the capacities of the facilities we need to place. We then establish a connection between the $m$-CFLP and a norm minimization problem in the Wasserstein space, which enables us to show that if the number of agent goes to infinity the limit of the ratio between the expected Social Cost of an ERM and the expected optimal Social Cost is finite and characterize its value. Noticeably, our method generalizes to encompass other truthful mechanisms and other metrics, such as the $l_p$ and Maximum costs. We conclude our theoretical analysis by characterizing the optimal ERM tailored to a $m$-CFLP and a distribution $μ$, that is the ERM whose limit Bayesian approximation ratio is the lowest compared to all other feasible ERMs. We consider mainly two frameworks: (i) in the first framework, the total facility capacity matches the number of agents, (ii) in the second framework, $m=2$. When we consider the Maximum Cost, we retrieve the optimal ERM for every $μ$, while for the Social Cost, we characterize the solution when the measure $μ$ is symmetric. Lastly, we numerically compare the performance of the ERMs against other truthful mechanisms and evaluate how quickly the Bayesian approximation ratio converges to its limit.

cs.GT↗

arXiv2024

Multimodal Radiomics Model for Predicting Gold Nanoparticles Accumulation in Mouse Tumors

Background: Nanoparticles can accumulate in solid tumors, serving as diagnostic or therapeutic agents for cancer. Clinical translation is challenging due to low accumulation in tumors and heterogeneity between tumor types and individuals. Tools to identify this heterogeneity and predict nanoparticle accumulation are needed. Advanced imaging techniques combined with radiomics and AI may offer a solution. Methods: 183 mice were used to create seven subcutaneous tumor models, with three sizes (15nm, 40nm, 70nm) of gold nanoparticles injected via the tail vein. Accumulation was measured using ICP-OES. Data were divided into training and test sets (7:3). Tumors were categorized into high and low uptake groups based on the median value of the training set. Before injection, multimodal imaging data (CT, B-mode ultrasound, SWE, CEUS) were acquired, and radiomics features extracted. LASSO and RFE algorithms built a radiomics signature. This, along with tumor type and mean values from CT and SWE, constructed the best model using SVM. For each tumor in the test set, the radiomics signature predicted gold nanoparticle uptake. Model performance was evaluated by AUC. Results: Significant variability in gold nanoparticle accumulation was observed among tumors (P < 0.001). The median accumulation in the training set was 3.37% ID/g. Nanoparticle size was not a main determinant of uptake (P > 0.05). The composite model based on radiomics signature outperformed the basic model in both training (AUC 0.93 vs. 0.68) and testing (0.78 vs. 0.61) datasets. Conclusion: The composite model identifies tumor heterogeneity and predicts high uptake of gold nanoparticles, improving patient stratification and supporting nanomedicine's clinical application.

q-bio.QM↗

arXiv2024

Refining Self-Supervised Learnt Speech Representation using Brain Activations

It was shown in literature that speech representations extracted by self-supervised pre-trained models exhibit similarities with brain activations of human for speech perception and fine-tuning speech representation models on downstream tasks can further improve the similarity. However, it still remains unclear if this similarity can be used to optimize the pre-trained speech models. In this work, we therefore propose to use the brain activations recorded by fMRI to refine the often-used wav2vec2.0 model by aligning model representations toward human neural responses. Experimental results on SUPERB reveal that this operation is beneficial for several downstream tasks, e.g., speaker verification, automatic speech recognition, intent classification.One can then consider the proposed method as a new alternative to improve self-supervised speech models.

eess.AS↗

arXiv2024

SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validation

Reliability in cloud AI infrastructure is crucial for cloud service providers, prompting the widespread use of hardware redundancies. However, these redundancies can inadvertently lead to hidden degradation, so called "gray failure", for AI workloads, significantly affecting end-to-end performance and concealing performance issues, which complicates root cause analysis for failures and regressions. We introduce SuperBench, a proactive validation system for AI infrastructure that mitigates hidden degradation caused by hardware redundancies and enhances overall reliability. SuperBench features a comprehensive benchmark suite, capable of evaluating individual hardware components and representing most real AI workloads. It comprises a Validator which learns benchmark criteria to clearly pinpoint defective components. Additionally, SuperBench incorporates a Selector to balance validation time and issue-related penalties, enabling optimal timing for validation execution with a tailored subset of benchmarks. Through testbed evaluation and simulation, we demonstrate that SuperBench can increase the mean time between incidents by up to 22.61x. SuperBench has been successfully deployed in Azure production, validating hundreds of thousands of GPUs over the last two years.

cs.DC↗

arXiv2024

LDM-SVC: Latent Diffusion Model Based Zero-Shot Any-to-Any Singing Voice Conversion with Singer Guidance

Any-to-any singing voice conversion (SVC) is an interesting audio editing technique, aiming to convert the singing voice of one singer into that of another, given only a few seconds of singing data. However, during the conversion process, the issue of timbre leakage is inevitable: the converted singing voice still sounds like the original singer's voice. To tackle this, we propose a latent diffusion model for SVC (LDM-SVC) in this work, which attempts to perform SVC in the latent space using an LDM. We pretrain a variational autoencoder structure using the noted open-source So-VITS-SVC project based on the VITS framework, which is then used for the LDM training. Besides, we propose a singer guidance training method based on classifier-free guidance to further suppress the timbre of the original singer. Experimental results show the superiority of the proposed method over previous works in both subjective and objective evaluations of timbre similarity.

eess.AS↗

arXiv2024

A Joint Optimization Approach for Power-Efficient Heterogeneous OFDMA Radio Access Networks

Heterogeneous networks have emerged as a popular solution for accommodating the growing number of connected devices and increasing traffic demands in cellular networks. While offering broader coverage, higher capacity, and lower latency, the escalating energy consumption poses sustainability challenges. In this paper a novel optimization approach for OFDMA heterogeneous networks is proposed to minimize transmission power while respecting individual users throughput constraints. The problem is formulated as a mixed integer geometric program, and optimizes at once multiple system variables such as user association, working bandwidth, and base stations transmission powers. Crucially, the proposed approach becomes a convex optimization problem when user-base station associations are provided. Evaluations in multiple realistic scenarios from the production mobile network of a major European operator and based on precise channel gains and throughput requirements from measured data validate the effectiveness of the proposed approach. Overall, our original solution paves the road for greener connectivity by reducing the energy footprint of heterogeneous mobile networks, hence fostering more sustainable communication systems.

math.OC↗

arXiv2024

A Study of the Latest Updates of the Readout System for the Hybird-Pixel Detector at HEPS

The High Energy Photon Source (HEPS) represents a fourth-generation light source. This facility has made unprecedented advancements in accelerator technology, necessitating the development of new detectors to satisfy physical requirements such as single-photon resolution, large dynamic range, and high frame rates. Since 2016, the Institute of High Energy Physics has introduced the first user-experimental hybrid pixel detector, progressing to the fourth-generation million-pixel detector designed for challenging conditions, with the dual-threshold single-photon detector HEPS-Beijing PIXel (HEPS-BPIX) set as the next-generation target. HEPS-BPIX will employ the entirely new Application-Specific Integrated Circuit (ASIC) BP40 for pixel information readout. Data flow will be managed and controlled through readout electronics based on a two-tier Field-Programmable Gate Array (FPGA) system: the Front-End Electronics (FEE) and the Input-Output Board (IOB) handle the fan-out for 12 ASICs, and the u4FCP is tasked with processing serial data on high-speed links, transferring pixel-level data to the back-end RTM and uTCA chassis, or independently outputting through a network port, enabling remote control of the entire detector. The new HEPS-BPIX firmware has undergone a comprehensive redesign and update to meet the electronic characteristics of the new chip and to improve the overall performance of the detector. We provide an overview of the core subunits of HEPS-BPIX, emphasizing the readout system, evaluating the new hardware and firmware, and highlighting some of its innovative features and characteristics.

physics.ins-det↗

arXiv2024

SIBO: A Simple Booster for Parameter-Efficient Fine-Tuning

Fine-tuning all parameters of large language models (LLMs) necessitates substantial computational power and extended time. Latest advancements in parameter-efficient fine-tuning (PEFT) techniques, such as Adapter tuning and LoRA, allow for adjustments to only a minor fraction of the parameters of these LLMs. Concurrently, it has been noted that the issue of over-smoothing diminishes the effectiveness of these Transformer-based LLMs, resulting in suboptimal performances in downstream tasks. In this paper, we present SIBO, which is a SImple BOoster to enhance PEFT, by injecting an initial residual. SIBO is straightforward and readily extensible to a range of state-of-the-art PEFT techniques to alleviate over-smoothing and enhance performance. Extensive experiments on 22 benchmark datasets demonstrate that SIBO significantly enhances the performance of various strong baselines, achieving up to 15.7% and 23.5% improvement over existing PEFT methods on the arithmetic and commonsense reasoning tasks, respectively.

cs.CL↗

arXiv2024

Federated Generative Learning with Foundation Models

Existing approaches in Federated Learning (FL) mainly focus on sending model parameters or gradients from clients to a server. However, these methods are plagued by significant inefficiency, privacy, and security concerns. Thanks to the emerging foundation generative models, we propose a novel federated learning framework, namely Federated Generative Learning. In this framework, each client can create text embeddings that are tailored to their local data, and send embeddings to the server. Then the informative training data can be synthesized remotely on the server using foundation generative models with these embeddings, which can benefit FL tasks. Our proposed framework offers several advantages, including increased communication efficiency, robustness to data heterogeneity, substantial performance improvements, and enhanced privacy protection. We validate these benefits through extensive experiments conducted on 12 datasets. For example, on the ImageNet100 dataset with a highly skewed data distribution, our method outperforms FedAvg by 12% in a single communication round, compared to FedAvg's performance over 200 communication rounds. We have released the code for all experiments conducted in this study.

cs.LG↗

arXiv2024

Large low-field magnetocaloric response in a ferromagnetic gadolinium orthophosphate

Bulk magnetic and thermodynamic measurements, along with mean-field calculations, were conducted on the ferromagnetic K3Gd5(PO4)6 powders. No magnetic ordering was observed until 2 K, while the application of an external field B > 1 T resulted in the splitting of the Gd3+ ground state multiplet and induced a non-cooperative Schottky effect. The average nearest-neighbor exchange strength |J1/kB| is determined to be 0.017 K, which leads to a remarkably large low field magnetic entropy change ΔSm = 36.2 J kg-1 K-1 under applied field change B = 2 T at temperature T = 2 K, as well as a maximum adiabatic temperature change Tad = 10.9 K. We contend that ferromagnetic gadolinium orthophosphates serve as a promising reservoir for exploring advanced magnetic refrigerants applicable under low magnetic fields.

cond-mat.mtrl-sci↗