arXiv subjects

Jie Chen

Publications and source records attributed to Jie Chen.

At least 289 records · Page 16Linked to original sources

arXiv2024

Make Your Actor Talk: Generalizable and High-Fidelity Lip Sync with Motion and Appearance Disentanglement

We aim to edit the lip movements in talking video according to the given speech while preserving the personal identity and visual details. The task can be decomposed into two sub-problems: (1) speech-driven lip motion generation and (2) visual appearance synthesis. Current solutions handle the two sub-problems within a single generative model, resulting in a challenging trade-off between lip-sync quality and visual details preservation. Instead, we propose to disentangle the motion and appearance, and then generate them one by one with a speech-to-motion diffusion model and a motion-conditioned appearance generation model. However, there still remain challenges in each stage, such as motion-aware identity preservation in (1) and visual details preservation in (2). Therefore, to preserve personal identity, we adopt landmarks to represent the motion, and further employ a landmark-based identity loss. To capture motion-agnostic visual details, we use separate encoders to encode the lip, non-lip appearance and motion, and then integrate them with a learned fusion module. We train MyTalk on a large-scale and diverse dataset. Experiments show that our method generalizes well to the unknown, even out-of-domain person, in terms of both lip sync and visual detail preservation. We encourage the readers to watch the videos on our project page (https://Ingrid789.github.io/MyTalk/).

cs.CV↗

arXiv2024

Data-driven Control Against False Data Injection Attacks

The rise of cyber-security concerns has brought significant attention to the analysis and design of cyber-physical systems (CPSs). Among the various types of cyberattacks, denial-of-service (DoS) attacks and false data injection (FDI) attacks can be easily launched and have become prominent threats. While resilient control against DoS attacks has received substantial research efforts, countermeasures developed against FDI attacks have been relatively limited, particularly when explicit system models are not available. To address this gap, the present paper focuses on the design of data-driven controllers for unknown linear systems subject to FDI attacks on the actuators, utilizing input-state data. To this end, a general FDI attack model is presented, which imposes minimally constraints on the switching frequency of attack channels and the magnitude of attack matrices. A dynamic state feedback control law is designed based on offline and online input-state data, which adapts to the channel switching of FDI attacks. This is achieved by solving two data-based semi-definite programs (SDPs) on-the-fly to yield a tight approximation of the set of subsystems consistent with both offline clean data and online attack-corrupted data. It is shown that under mild conditions on the attack, the proposed SDPs are recursively feasible and controller achieves exponential stability. Numerical examples showcase its effectiveness in mitigating the impact of FDI attacks.

eess.SY↗

arXiv2024

Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data

Large Language Models (LLMs) have shown excellent performance in language understanding, text generation, code synthesis, and many other tasks, while they still struggle in complex multi-step reasoning problems, such as mathematical reasoning. In this paper, through a newly proposed arithmetical puzzle problem, we show that the model can perform well on multi-step reasoning tasks via fine-tuning on high-quality synthetic data. Experimental results with the open-llama-3B model on three different test datasets show that not only the model can reach a zero-shot pass@1 at 0.44 on the in-domain dataset, it also demonstrates certain generalization capabilities on the out-of-domain datasets. Specifically, this paper has designed two out-of-domain datasets in the form of extending the numerical range and the composing components of the arithmetical puzzle problem separately. The fine-tuned models have shown encouraging performance on these two far more difficult tasks with the zero-shot pass@1 at 0.33 and 0.35, respectively.

cs.CL↗

arXiv2024

Representing Molecules as Random Walks Over Interpretable Grammars

Recent research in molecular discovery has primarily been devoted to small, drug-like molecules, leaving many similarly important applications in material design without adequate technology. These applications often rely on more complex molecular structures with fewer examples that are carefully designed using known substructures. We propose a data-efficient and interpretable model for representing and reasoning over such molecules in terms of graph grammars that explicitly describe the hierarchical design space featuring motifs to be the design basis. We present a novel representation in the form of random walks over the design space, which facilitates both molecule generation and property prediction. We demonstrate clear advantages over existing methods in terms of performance, efficiency, and synthesizability of predicted molecules, and we provide detailed insights into the method's chemical interpretability.

cs.LG↗

arXiv2024

Correction for the Weakening Magnetic Field within the Sunspot Umbra Observed by ASO-S/FMG

The magnetic field inside the sunspot umbra, as observed by the Full-disk MagnetoGraph (FMG) onboard the Advanced Space based Solar Observatory (ASO-S), was found to be experiencing a weakening. To address this issue, we employed a method developed by Xu et al. (2021) to correct the weakening in the data of 20 active regions observed by FMG during the period spanning December 29, 2022, to July 23, 2023. Research has revealed that the onset of magnetic field weakening occurs at a minimum magnetic field strength of 705 G, with the peak strength reaching up to 1931 G. We computed the change ratio (R1) of the unsigned magnetic flux within the sunspot umbra, considering measurements both before and after correction. The change ratio (R1) spans from 26% to 124%, indicating a significant increase in the unsigned magnetic flux within sunspot umbrae observed by FMG after correction. To illustrate this, we selected four active regions for comparison with data from the Helioseismic and Magnetic Imager (HMI). After correction, it is found that the unsigned magnetic flux in sunspot umbrae measured by FMG aligns more closely with that of HMI. This supports the effectiveness of the corrective method for FMG, despite imperfections, particularly at the umbra-penumbra boundary.

astro-ph.SR↗

arXiv2024

Observation of a large-scale filament eruption initiated by two small-scale erupting filaments pushing out from below

Filament eruptions often result in flares and coronal mass ejections (CMEs). Most studies attribute the filament eruptions to their instabilities or magnetic reconnection. In this study, we report a unique observation of a filament eruption whose initiation process has not been reported before. This large-scale filament, with a length of about 360 Mm crossing an active region, is forced to erupted by two small-scale erupting filaments pushing out from below. This process of multi-filament eruption results in an M6.4 flare in the active region NOAA 13229 on 25th February 2023. The whole process can be divided into three stages: the eruptions of two active-region filaments F1 and F2; the interactions between the erupting F1, F2, and the large-scale filament F3; and the eruption of F3. Though this multi-filament eruption occurs near the northwest limb of the solar disk, it produces a strong halo CME that causes a significant geomagnetic disturbance. Our observations present a new filament eruption mechanism, in which the initial kinetic energy of the eruption is obtained from and transported to by other erupting structures. This event provides us a unique insight into the dynamics of multi-filament eruptions and their corresponding effects on the interplanetary space.

astro-ph.SR↗

arXiv2024

Boundary Exploration for Bayesian Optimization With Unknown Physical Constraints

Bayesian optimization has been successfully applied to optimize black-box functions where the number of evaluations is severely limited. However, in many real-world applications, it is hard or impossible to know in advance which designs are feasible due to some physical or system limitations. These issues lead to an even more challenging problem of optimizing an unknown function with unknown constraints. In this paper, we observe that in such scenarios optimal solution typically lies on the boundary between feasible and infeasible regions of the design space, making it considerably more difficult than that with interior optima. Inspired by this observation, we propose BE-CBO, a new Bayesian optimization method that efficiently explores the boundary between feasible and infeasible designs. To identify the boundary, we learn the constraints with an ensemble of neural networks that outperform the standard Gaussian Processes for capturing complex boundaries. Our method demonstrates superior performance against state-of-the-art methods through comprehensive experiments on synthetic and real-world benchmarks. Code available at: https://github.com/yunshengtian/BE-CBO

cs.LG↗

arXiv2024

HARIS: Human-Like Attention for Reference Image Segmentation

Referring image segmentation (RIS) aims to locate the particular region corresponding to the language expression. Existing methods incorporate features from different modalities in a \emph{bottom-up} manner. This design may get some unnecessary image-text pairs, which leads to an inaccurate segmentation mask. In this paper, we propose a referring image segmentation method called HARIS, which introduces the Human-Like Attention mechanism and uses the parameter-efficient fine-tuning (PEFT) framework. To be specific, the Human-Like Attention gets a \emph{feedback} signal from multi-modal features, which makes the network center on the specific objects and discard the irrelevant image-text pairs. Besides, we introduce the PEFT framework to preserve the zero-shot ability of pre-trained encoders. Extensive experiments on three widely used RIS benchmarks and the PhraseCut dataset demonstrate that our method achieves state-of-the-art performance and great zero-shot ability.

cs.CV↗

arXiv2024

Prospective Role of Foundation Models in Advancing Autonomous Vehicles

With the development of artificial intelligence and breakthroughs in deep learning, large-scale Foundation Models (FMs), such as GPT, Sora, etc., have achieved remarkable results in many fields including natural language processing and computer vision. The application of FMs in autonomous driving holds considerable promise. For example, they can contribute to enhancing scene understanding and reasoning. By pre-training on rich linguistic and visual data, FMs can understand and interpret various elements in a driving scene, and provide cognitive reasoning to give linguistic and action instructions for driving decisions and planning. Furthermore, FMs can augment data based on the understanding of driving scenarios to provide feasible scenes of those rare occurrences in the long tail distribution that are unlikely to be encountered during routine driving and data collection. The enhancement can subsequently lead to improvement in the accuracy and reliability of autonomous driving systems. Another testament to the potential of FMs' applications lies in World Models, exemplified by the DREAMER series, which showcases the ability to comprehend physical laws and dynamics. Learning from massive data under the paradigm of self-supervised learning, World Model can generate unseen yet plausible driving environments, facilitating the enhancement in the prediction of road users' behaviors and the off-line training of driving strategies. In this paper, we synthesize the applications and future trends of FMs in autonomous driving. By utilizing the powerful capabilities of FMs, we strive to tackle the potential issues stemming from the long-tail distribution in autonomous driving, consequently advancing overall safety in this domain.

cs.CV↗

arXiv2024

GraCo: Granularity-Controllable Interactive Segmentation

Interactive Segmentation (IS) segments specific objects or parts in the image according to user input. Current IS pipelines fall into two categories: single-granularity output and multi-granularity output. The latter aims to alleviate the spatial ambiguity present in the former. However, the multi-granularity output pipeline suffers from limited interaction flexibility and produces redundant results. In this work, we introduce Granularity-Controllable Interactive Segmentation (GraCo), a novel approach that allows precise control of prediction granularity by introducing additional parameters to input. This enhances the customization of the interactive system and eliminates redundancy while resolving ambiguity. Nevertheless, the exorbitant cost of annotating multi-granularity masks and the lack of available datasets with granularity annotations make it difficult for models to acquire the necessary guidance to control output granularity. To address this problem, we design an any-granularity mask generator that exploits the semantic property of the pre-trained IS model to automatically generate abundant mask-granularity pairs without requiring additional manual annotation. Based on these pairs, we propose a granularity-controllable learning strategy that efficiently imparts the granularity controllability to the IS model. Extensive experiments on intricate scenarios at object and part levels demonstrate that our GraCo has significant advantages over previous methods. This highlights the potential of GraCo to be a flexible annotation tool, capable of adapting to diverse segmentation scenarios. The project page: https://zhao-yian.github.io/GraCo.

cs.CV↗

arXiv2024

Generation of Uncorrelated Residual Variables for Chemical Process Fault Diagnosis via Transfer Learning-based Input-Output Decoupled Network

Structural decoupling has played an essential role in model-based fault isolation and estimation in past decades, which facilitates accurate fault localization and reconstruction thanks to the diagonal transfer matrix design. However, traditional methods exhibit limited effectiveness in modeling high-dimensional nonlinearity and big data, and the decoupling idea has not been well-valued in data-driven frameworks. Known for big data and complex feature extraction capabilities, deep learning has recently been used to develop residual generation models. Nevertheless, it lacks decoupling-related diagnostic designs. To this end, this paper proposes a transfer learning-based input-output decoupled network (TDN) for diagnostic purposes, which consists of an input-output decoupled network (IDN) and a pre-trained variational autocoder (VAE). In IDN, uncorrelated residual variables are generated by diagonalization and parallel computing operations. During the transfer learning phase, knowledge of normal status is provided according to VAE's loss and maximum mean discrepancy loss to guide the training of IDN. After training, IDN learns the mapping from faulty to normal, thereby serving as the fault detection index and the estimated fault signal simultaneously. At last, the effectiveness of the developed TDN is verified by a numerical example and a chemical simulation.

cs.LG↗

arXiv2024

Adaptive Catalyst Discovery Using Multicriteria Bayesian Optimization with Representation Learning

High-performance catalysts are crucial for sustainable energy conversion and human health. However, the discovery of catalysts faces challenges due to the absence of efficient approaches to navigating vast and high-dimensional structure and composition spaces. In this study, we propose a high-throughput computational catalyst screening approach integrating density functional theory (DFT) and Bayesian Optimization (BO). Within the BO framework, we propose an uncertainty-aware atomistic machine learning model, UPNet, which enables automated representation learning directly from high-dimensional catalyst structures and achieves principled uncertainty quantification. Utilizing a constrained expected improvement acquisition function, our BO framework simultaneously considers multiple evaluation criteria. Using the proposed methods, we explore catalyst discovery for the CO2 reduction reaction. The results demonstrate that our approach achieves high prediction accuracy, facilitates interpretable feature extraction, and enables multicriteria design optimization, leading to significant reduction of computing power and time (10x reduction of required DFT calculations) in high-performance catalyst discovery.

cs.LG↗

arXiv2024

One-Step Abductive Multi-Target Learning with Diverse Noisy Samples and Its Application to Tumour Segmentation for Breast Cancer

Recent studies have demonstrated the effectiveness of the combination of machine learning and logical reasoning, including data-driven logical reasoning, knowledge driven machine learning and abductive learning, in inventing advanced technologies for different artificial intelligence applications. One-step abductive multi-target learning (OSAMTL), an approach inspired by abductive learning, via simply combining machine learning and logical reasoning in a one-step balanced multi-target learning way, has as well shown its effectiveness in handling complex noisy labels of a single noisy sample in medical histopathology whole slide image analysis (MHWSIA). However, OSAMTL is not suitable for the situation where diverse noisy samples (DiNS) are provided for a learning task. In this paper, giving definition of DiNS, we propose one-step abductive multi-target learning with DiNS (OSAMTL-DiNS) to expand the original OSAMTL to handle complex noisy labels of DiNS. Applying OSAMTL-DiNS to tumour segmentation for breast cancer in MHWSIA, we show that OSAMTL-DiNS is able to enable various state-of-the-art approaches for learning from noisy labels to achieve more rational predictions. We released a model pre-trained with OSAMTL-DiNS for tumour segmentation in HE-stained pre-treatment biopsy images in breast cancer, which has been successfully applied as a pre-processing tool to extract tumour-associated stroma compartment for predicting the pathological complete response to neoadjuvant chemotherapy in breast cancer.

cs.LG↗

arXiv2024

Evolutionary game on any hypergraph

Cooperation plays a fundamental role in societal and biological domains, and the population structure profoundly shapes the dynamics of evolution. Practically, individuals behave either altruistically or egoistically in multiple groups, such as relatives, friends and colleagues, and feedbacks from these groupwise interactions will contribute to one's cognition and behavior. Due to the intricacy within and between groups, exploration of evolutionary dynamics over hypergraphs is relatively limited to date. To uncover this conundrum, we develop a higher-order random walk framework for five distinct updating rules, thus establishing explicit conditions for cooperation emergence on hypergraphs, and finding the overlaps between groups tend to foster cooperative behaviors. Our systematic analysis quantifies how the order and hyperdegree govern evolutionary outcomes. We also discover that whenever following a group wisdom update protocol, choosing a high-fitness group to interact equally within its members, cooperators will significantly prevail throughout the community. These findings underscore a crucial role of higher-order interaction and interdisciplinary collaboration throughout a broad range of living systems, favoring social prosperity.

nlin.AO↗

arXiv2024

DETRs Beat YOLOs on Real-time Object Detection

The YOLO series has become the most popular framework for real-time object detection due to its reasonable trade-off between speed and accuracy. However, we observe that the speed and accuracy of YOLOs are negatively affected by the NMS. Recently, end-to-end Transformer-based detectors (DETRs) have provided an alternative to eliminating NMS. Nevertheless, the high computational cost limits their practicality and hinders them from fully exploiting the advantage of excluding NMS. In this paper, we propose the Real-Time DEtection TRansformer (RT-DETR), the first real-time end-to-end object detector to our best knowledge that addresses the above dilemma. We build RT-DETR in two steps, drawing on the advanced DETR: first we focus on maintaining accuracy while improving speed, followed by maintaining speed while improving accuracy. Specifically, we design an efficient hybrid encoder to expeditiously process multi-scale features by decoupling intra-scale interaction and cross-scale fusion to improve speed. Then, we propose the uncertainty-minimal query selection to provide high-quality initial queries to the decoder, thereby improving accuracy. In addition, RT-DETR supports flexible speed tuning by adjusting the number of decoder layers to adapt to various scenarios without retraining. Our RT-DETR-R50 / R101 achieves 53.1% / 54.3% AP on COCO and 108 / 74 FPS on T4 GPU, outperforming previously advanced YOLOs in both speed and accuracy. We also develop scaled RT-DETRs that outperform the lighter YOLO detectors (S and M models). Furthermore, RT-DETR-R50 outperforms DINO-R50 by 2.2% AP in accuracy and about 21 times in FPS. After pre-training with Objects365, RT-DETR-R50 / R101 achieves 55.3% / 56.2% AP. The project page: https://zhao-yian.github.io/RTDETR.

cs.CV↗

arXiv2024

Consistency Enhancement-Based Deep Multiview Clustering via Contrastive Learning

Multiview clustering (MVC) segregates data samples into meaningful clusters by synthesizing information across multiple views. Moreover, deep learning-based methods have demonstrated their strong feature learning capabilities in MVC scenarios. However, effectively generalizing feature representations while maintaining consistency is still an intractable problem. In addition, most existing deep clustering methods based on contrastive learning overlook the consistency of the clustering representations during the clustering process. In this paper, we show how the above problems can be overcome and propose a consistent enhancement-based deep MVC method via contrastive learning (CCEC). Specifically, semantic connection blocks are incorporated into a feature representation to preserve the consistent information among multiple views. Furthermore, the representation process for clustering is enhanced through spectral clustering, and the consistency across multiple views is improved. Experiments conducted on five datasets demonstrate the effectiveness and superiority of our method in comparison with the state-of-the-art (SOTA) methods. The code for this method can be accessed at https://anonymous.4open.science/r/CCEC-E84E/.

cs.LG↗

arXiv2024

Exotic magnetism in perovskite KOsO3

A new perovskite KOsO3 has been stabilized under high-pressure and high temperature conditions. It is cubic at 500 K (Pm-3m) and undergoes subsequent phase transitions to tetragonal at 320 K (P4/mmm) and rhombohedral (R-3m) at 230 K as shown from refining synchrotron X-ray powder diffraction (SXRD) data. The larger orbital overlap integral and the extended wavefunction of 5d electrons in the perovskite KOsO3 allow to explore physics from the regime where Mott and Hund's rule couplings dominate to the state where the multiple interactions are on equal footing. We demonstrate an exotic magnetic ordering phase found by neutron powder diffraction along with physical properties via a suite of measurements including magnetic and transport properties, differential scanning calorimetry, and specific heat, which provide comprehensive information for a system at the crossover from localized to itinerant electronic behavior.

cond-mat.str-el↗

arXiv2024

Attitude Tracking of Uncertain Flexible Spacecraft Systems Subject to Unknown External Disturbances

In this paper, we investigate the attitude tracking problem of uncertain flexible spacecraft systems subject to external disturbances. In sharp contrast to existing results, the dynamics of flexible spacecraft systems and external disturbances are allowed to be unknown. To deal with the challenges by these unknown factors, we develop a class of nonlinear internal models which converts the attitude tracking problem of uncertain flexible spacecraft systems into a regulation problem of an augmented system. Furthermore, to overcome the difficulties caused by the unmeasurable modal variable, the uncertainty introduced by the internal model, and the cross-coupling of the uncertainties with the system state, we design an auxiliary dynamic system for auxiliary stabilization, a dynamic compensator for dynamic compensation, and a linearly parameterized transformation for adaptive regulation in sequence. By introducing a series of coordinate and input transformations, we propose an adaptive dynamic control law to achieve regulation of the augmented system and thus leading to the solution to the attitude tracking problem. In addition, we analyze the convergence issue of the estimated parameter to its true value by the persistently exciting condition. Finally, the effec tiveness of the developed approach is verified by its application to the attitude manoeuvre of a flexible spacecraft system in the presence of external disturbances.

math.OC↗