arXiv subjects

Jin Wang

Publications and source records attributed to Jin Wang.

At least 217 records · Page 12Linked to original sources

arXiv2024

Dissipation-driven quantum generative adversarial networks

Quantum machine learning holds the promise of harnessing quantum advantage to achieve speedup beyond classical algorithms. Concurrently, research indicates that dissipation can serve as an effective resource in quantum computation. In this paper, we introduce a novel dissipation-driven quantum generative adversarial network (DQGAN) architecture specifically tailored for generating classical data. Our DQGAN comprises two interacting networks: a generative network and a discriminative network, both constructed from qubits. The classical data is encoded into the input qubits of the input layer via strong tailored dissipation processes. This encoding scheme enables us to extract both the generated data and the classification results by measuring the observables of the steady state of the output qubits. The network coupling weight, i.e., the strength of the interaction Hamiltonian between layers, is iteratively updated during the training process. This training procedure closely resembles the training of conventional generative adversarial networks (GANs). By alternately updating the two networks, we foster adversarial learning until the equilibrium point is reached. Our preliminary numerical test on a simplified instance of the task substantiate the feasibility of our DQGAN model.

quant-ph↗

arXiv2024

LLM-assisted Labeling Function Generation for Semantic Type Detection

Detecting semantic types of columns in data lake tables is an important application. A key bottleneck in semantic type detection is the availability of human annotation due to the inherent complexity of data lakes. In this paper, we propose using programmatic weak supervision to assist in annotating the training data for semantic type detection by leveraging labeling functions. One challenge in this process is the difficulty of manually writing labeling functions due to the large volume and low quality of the data lake table datasets. To address this issue, we explore employing Large Language Models (LLMs) for labeling function generation and introduce several prompt engineering strategies for this purpose. We conduct experiments on real-world web table datasets. Based on the initial results, we perform extensive analysis and provide empirical insights and future directions for researchers in this field.

cs.DB↗

arXiv2024

MegActor-$Σ$: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer

Diffusion models have demonstrated superior performance in the field of portrait animation. However, current approaches relied on either visual or audio modality to control character movements, failing to exploit the potential of mixed-modal control. This challenge arises from the difficulty in balancing the weak control strength of audio modality and the strong control strength of visual modality. To address this issue, we introduce MegActor-$Σ$: a mixed-modal conditional diffusion transformer (DiT), which can flexibly inject audio and visual modality control signals into portrait animation. Specifically, we make substantial advancements over its predecessor, MegActor, by leveraging the promising model structure of DiT and integrating audio and visual conditions through advanced modules within the DiT framework. To further achieve flexible combinations of mixed-modal control signals, we propose a ``Modality Decoupling Control" training strategy to balance the control strength between visual and audio modalities, along with the ``Amplitude Adjustment" inference strategy to freely regulate the motion amplitude of each modality. Finally, to facilitate extensive studies in this field, we design several dataset evaluation metrics to filter out public datasets and solely use this filtered dataset to train MegActor-$Σ$. Extensive experiments demonstrate the superiority of our approach in generating vivid portrait animations, outperforming previous methods trained on private dataset.

cs.CV↗

arXiv2024

A proof of contribution in blockchain using game theoretical deep learning model

Building elastic and scalable edge resources is an inevitable prerequisite for providing platform-based smart city services. Smart city services are delivered through edge computing to provide low-latency applications. However, edge computing has always faced the challenge of limited resources. A single edge device cannot undertake the various intelligent computations in a smart city, and the large-scale deployment of edge devices from different service providers to build an edge resource platform has become a necessity. Selecting computing power from different service providers is a game-theoretic problem. To incentivize service providers to actively contribute their valuable resources and provide low-latency collaborative computing power, we introduce a game-theoretic deep learning model to reach a consensus among service providers on task scheduling and resource provisioning. Traditional centralized resource management approaches are inefficient and lack credibility, while the introduction of blockchain technology can enable decentralized resource trading and scheduling. We propose a contribution-based proof mechanism to provide the low-latency service of edge computing. The deep learning model consists of dual encoders and a single decoder, where the GNN (Graph Neural Network) encoder processes structured decision action data, and the RNN (Recurrent Neural Network) encoder handles time-series task scheduling data. Extensive experiments have demonstrated that our model reduces latency by 584% compared to the state-of-the-art.

cs.CR↗

arXiv2024

AEMLO: AutoEncoder-Guided Multi-Label Oversampling

Class imbalance significantly impacts the performance of multi-label classifiers. Oversampling is one of the most popular approaches, as it augments instances associated with less frequent labels to balance the class distribution. Existing oversampling methods generate feature vectors of synthetic samples through replication or linear interpolation and assign labels through neighborhood information. Linear interpolation typically generates new samples between existing data points, which may result in insufficient diversity of synthesized samples and further lead to the overfitting issue. Deep learning-based methods, such as AutoEncoders, have been proposed to generate more diverse and complex synthetic samples, achieving excellent performance on imbalanced binary or multi-class datasets. In this study, we introduce AEMLO, an AutoEncoder-guided Oversampling technique specifically designed for tackling imbalanced multi-label data. AEMLO is built upon two fundamental components. The first is an encoder-decoder architecture that enables the model to encode input data into a low-dimensional feature space, learn its latent representations, and then reconstruct it back to its original dimension, thus applying to the generation of new data. The second is an objective function tailored to optimize the sampling task for multi-label scenarios. We show that AEMLO outperforms the existing state-of-the-art methods with extensive empirical studies.

cs.LG↗

arXiv2024

Chinese Metaphor Recognition Using a Multi-stage Prompting Large Language Model

Metaphors are common in everyday language, and the identification and understanding of metaphors are facilitated by models to achieve a better understanding of the text. Metaphors are mainly identified and generated by pre-trained models in existing research, but situations, where tenors or vehicles are not included in the metaphor, cannot be handled. The problem can be effectively solved by using Large Language Models (LLMs), but significant room for exploration remains in this early-stage research area. A multi-stage generative heuristic-enhanced prompt framework is proposed in this study to enhance the ability of LLMs to recognize tenors, vehicles, and grounds in Chinese metaphors. In the first stage, a small model is trained to obtain the required confidence score for answer candidate generation. In the second stage, questions are clustered and sampled according to specific rules. Finally, the heuristic-enhanced prompt needed is formed by combining the generated answer candidates and demonstrations. The proposed model achieved 3rd place in Track 1 of Subtask 1, 1st place in Track 2 of Subtask 1, and 1st place in both tracks of Subtask 2 at the NLPCC-2024 Shared Task 9.

cs.CL↗

arXiv2024

Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model

Enabling humanoid robots to perform autonomously loco-manipulation in unstructured environments is crucial and highly challenging for achieving embodied intelligence. This involves robots being able to plan their actions and behaviors in long-horizon tasks while using multi-modality to perceive deviations between task execution and high-level planning. Recently, large language models (LLMs) have demonstrated powerful planning and reasoning capabilities for comprehension and processing of semantic information through robot control tasks, as well as the usability of analytical judgment and decision-making for multi-modal inputs. To leverage the power of LLMs towards humanoid loco-manipulation, we propose a novel language-model based framework that enables robots to autonomously plan behaviors and low-level execution under given textual instructions, while observing and correcting failures that may occur during task execution. To systematically evaluate this framework in grounding LLMs, we created the robot 'action' and 'sensing' behavior library for task planning, and conducted mobile manipulation tasks and experiments in both simulated and real environments using the CENTAURO robot, and verified the effectiveness and application of this approach in robotic tasks with autonomous behavioral planning.

cs.RO↗

arXiv2024

Mpemba effects in nonequilibrium open quantum systems

The Mpemba effect was originally referred to as the faster icing of a higher-temperature system than a lower-temperature system, and was later generalized to anomalous decays of both classical and quantum observables to equilibrium states. Mpemba effect is mostly considered in classical systems and during cooling processes towards equilibrium states. We investigate the emergence of the effect in nonequilibrium quantum systems where the system has no asymptotic equilibrium state to approach. Instead, the system is put in contact with two different baths, and only a nonequilibrium state exists, sustained by constant energy injection from the surrounding thermal baths. Firstly, we show that the nonequilibrium conditions can dramatically enlarge the parameter regimes where the MPE emerges. Secondly, we demonstrate that the anomalous MPEs and inverse MPEs emerge in the evolution of quantum correlations in the two-site fermionic system and that nonequilibrium conditions can expedite or delay the MPEs. Thirdly, we show that the nonequilibrium-induced quantum coherence can have considerable contributions to the emergence of the MPE which the conventional Lindbladian dynamics fails to capture.

quant-ph↗

arXiv2024

MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

The capability to process multiple images is crucial for Large Vision-Language Models (LVLMs) to develop a more thorough and nuanced understanding of a scene. Recent multi-image LVLMs have begun to address this need. However, their evaluation has not kept pace with their development. To fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess LVLMs across a wide range of multi-image tasks. MMIU encompasses 7 types of multi-image relationships, 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions, making it the most extensive benchmark of its kind. Our evaluation of 24 popular LVLMs, including both open-source and proprietary models, reveals significant challenges in multi-image comprehension, particularly in tasks involving spatial understanding. Even the most advanced models, such as GPT-4o, achieve only 55.7% accuracy on MMIU. Through multi-faceted analytical experiments, we identify key performance gaps and limitations, providing valuable insights for future model and data improvements. We aim for MMIU to advance the frontier of LVLM research and development, moving us toward achieving sophisticated multimodal multi-image user interactions.

cs.CV↗

arXiv2024

AFGI: Towards Accurate and Fast-convergent Gradient Inversion Attack in Federated Learning

Federated learning (FL) empowers privacypreservation in model training by only exposing users' model gradients. Yet, FL users are susceptible to gradient inversion attacks (GIAs) which can reconstruct ground-truth training data such as images based on model gradients. However, reconstructing high-resolution images by existing GIAs faces two challenges: inferior accuracy and slow-convergence, especially when duplicating labels exist in the training batch. To address these challenges, we present an Accurate and Fast-convergent Gradient Inversion attack algorithm, called AFGI, with two components: Label Recovery Block (LRB) which can accurately restore duplicating labels of private images based on exposed gradients; VME Regularization Term, which includes the total variance of reconstructed images, the discrepancy between three-channel means and edges, between values from exposed gradients and reconstructed images, respectively. The AFGI can be regarded as a white-box attack strategy to reconstruct images by leveraging labels recovered by LRB. In particular, AFGI is efficient that accurately reconstruct ground-truth images when users' training batch size is up to 48. Our experimental results manifest that AFGI can diminish 85% time costs while achieving superb inversion quality in the ImageNet dataset. At last, our study unveils the shortcomings of FL in privacy-preservation, prompting the development of more advanced countermeasure strategies.

cs.CV↗

arXiv2024

Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation

Zero-shot dialogue state tracking (DST) seeks to enable dialogue systems to transition to unfamiliar domains without manual annotation or extensive retraining. Prior research has approached this objective by embedding prompts into language models (LMs). Common methodologies include integrating prompts at the input layer or introducing learnable variables at each transformer layer. Nonetheless, each strategy exhibits inherent limitations. Prompts integrated at the input layer risk underutilization, with their impact potentially diminishing across successive transformer layers. Conversely, the addition of learnable variables to each layer can complicate the training process and increase inference latency. To tackle the issues mentioned above, this paper proposes Dual Low-Rank Adaptation (DualLoRA), a plug-and-play architecture designed for zero-shot DST. DualLoRA incorporates two distinct Low-Rank Adaptation (LoRA) components, targeting both dialogue context processing and prompt optimization, to ensure the comprehensive influence of prompts throughout the transformer model layers. This is achieved without incurring additional inference latency, showcasing an efficient integration into existing architectures. Through rigorous evaluation on the MultiWOZ and SGD datasets, DualLoRA demonstrates notable improvements across multiple domains, outperforming traditional baseline methods in zero-shot settings. Our code is accessible at: \url{https://github.com/suntea233/DualLoRA}.

cs.CL↗

arXiv2024

Nonequilibrium dynamics and thermodynamics provide the underlying physical mechanism of the perceptual rivalry

Perceptual rivalry, where conflicting sensory information leads to alternating perceptions crucial for associated cognitive function, has attracted researcher's attention for long. Despite progresses being made, recent studies have revealed limitations and inconsistencies in our understanding across various rivalry contexts. We develop a unified physical framework, where perception undergoes a consecutive phase transition process encompassing different multi-state competitions. We reveal the underlying mechanisms of perceptual rivalry by identifying dominant switching paths among perceptual states and quantifying mean perceptual durations, switching frequencies, and proportions of different perceptions. We uncover the underlying nonequilibrium dynamics and thermodynamics by analyzing average nonequilibrium flux and entropy production rate, while associated time series irreversibility reflects the underlying nonequilibrium mechanism of perceptual rivalry and link thermodynamical results with neuro-electrophysiological experiments. Our framework provides a global and physical understanding of brain perception, which may go beyond cognitive science or psychology but embodies the connection with wider fields as decision-making.

q-bio.NC↗

arXiv2024

PORCA: Root Cause Analysis with Partially Observed Data

Root Cause Analysis (RCA) aims at identifying the underlying causes of system faults by uncovering and analyzing the causal structure from complex systems. It has been widely used in many application domains. Reliable diagnostic conclusions are of great importance in mitigating system failures and financial losses. However, previous studies implicitly assume a full observation of the system, which neglect the effect of partial observation (i.e., missing nodes and latent malfunction). As a result, they fail in deriving reliable RCA results. In this paper, we unveil the issues of unobserved confounders and heterogeneity in partial observation and come up with a new problem of root cause analysis with partially observed data. To achieve this, we propose PORCA, a novel RCA framework which can explore reliable root causes under both unobserved confounders and unobserved heterogeneity. PORCA leverages magnified score-based causal discovery to efficiently optimize acyclic directed mixed graph under unobserved confounders. In addition, we also develop a heterogeneity-aware scheduling strategy to provide adaptive sample weights. Extensive experimental results on one synthetic and two real-world datasets demonstrate the effectiveness and superiority of the proposed framework.

cs.AI↗

arXiv2024

Probing the interaction energy of two $^{85}$Rb atoms in an optical tweezer via spin-motion coupling

The inherent polarization gradients in tight optical tweezers can be used to couple the atomic spins to the two-body motion under the action of a microwave spin-flip transition, so that such a spin-motion coupling offers an important control knob on the motional states of optically trapped two colliding atoms. Here, after preparing two elastically scattering $^{85}$Rb atoms in the three-dimensional ground-state in the optical tweezer, we employed this control in order to probe the colliding energies of elastic and inelastic channels. The combination of microwave spectra and corresponding s-wave pseudopotential model allows us to infer the effect of the state-dependent trapping potentials on the elastic colliding energies, as well as to reveal how the presence of inelastic interactions affects elastic part of the relative potential. Our work shows that the spin-motion coupling in a tight optical tweezer expand the experimental toolbox for fundamental studies of ultracold collisions in the two body systems with reactive collisions, and potentially for that of more complex interactions, such as optically trapped atom-molecule and molecule-molecule interactions.

physics.atom-ph↗

arXiv2024

HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation

Enabling robots to autonomously perform hybrid motions in diverse environments can be beneficial for long-horizon tasks such as material handling, household chores, and work assistance. This requires extensive exploitation of intrinsic motion capabilities, extraction of affordances from rich environmental information, and planning of physical interaction behaviors. Despite recent progress has demonstrated impressive humanoid whole-body control abilities, they struggle to achieve versatility and adaptability for new tasks. In this work, we propose HYPERmotion, a framework that learns, selects and plans behaviors based on tasks in different scenarios. We combine reinforcement learning with whole-body optimization to generate motion for 38 actuated joints and create a motion library to store the learned skills. We apply the planning and reasoning features of the large language models (LLMs) to complex loco-manipulation tasks, constructing a hierarchical task graph that comprises a series of primitive behaviors to bridge lower-level execution with higher-level planning. By leveraging the interaction of distilled spatial geometry and 2D observation with a visual language model (VLM) to ground knowledge into a robotic morphology selector to choose appropriate actions in single- or dual-arm, legged or wheeled locomotion. Experiments in simulation and real-world show that learned motions can efficiently adapt to new tasks, demonstrating high autonomy from free-text commands in unstructured scenes. Videos and website: hy-motion.github.io/

cs.RO↗

arXiv2024

On the fidelity of information retrieval in the black hole final state model with scrambling interactions

We study the fidelity of information retrieval in the black hole final state model by taking into account the interactions between the collapsing matter and the infalling Hawking radiation inside the event horizon. By utilizing a scrambling unitary operator to model these interactions, our direct calculations suggest that the information is almost lost during the process of black hole evaporation. We then proceed to employ the Yoshida-Kitaev decoding strategy for information retrieval. Although we observe a finite improvement in decoding fidelity, it does not reach unity. This indicates that the issue of unitarity in black hole evaporation may not be fully resolved within the framework of the final state model.

hep-th↗

arXiv2024

A Blueprint Architecture of Compound AI Systems for Enterprise

Large Language Models (LLMs) have showcased remarkable capabilities surpassing conventional NLP challenges, creating opportunities for use in production use cases. Towards this goal, there is a notable shift to building compound AI systems, wherein LLMs are integrated into an expansive software infrastructure with many components like models, retrievers, databases and tools. In this paper, we introduce a blueprint architecture for compound AI systems to operate in enterprise settings cost-effectively and feasibly. Our proposed architecture aims for seamless integration with existing compute and data infrastructure, with ``stream'' serving as the key orchestration concept to coordinate data and instructions among agents and other components. Task and data planners, respectively, break down, map, and optimize tasks and data to available agents and data sources defined in respective registries, given production constraints such as accuracy and latency.

cs.DB↗

arXiv2024

Towards RGB-NIR Cross-modality Image Registration and Beyond

This paper focuses on the area of RGB(visible)-NIR(near-infrared) cross-modality image registration, which is crucial for many downstream vision tasks to fully leverage the complementary information present in visible and infrared images. In this field, researchers face two primary challenges - the absence of a correctly-annotated benchmark with viewpoint variations for evaluating RGB-NIR cross-modality registration methods and the problem of inconsistent local features caused by the appearance discrepancy between RGB-NIR cross-modality images. To address these challenges, we first present the RGB-NIR Image Registration (RGB-NIR-IRegis) benchmark, which, for the first time, enables fair and comprehensive evaluations for the task of RGB-NIR cross-modality image registration. Evaluations of previous methods highlight the significant challenges posed by our RGB-NIR-IRegis benchmark, especially on RGB-NIR image pairs with viewpoint variations. To analyze the causes of the unsatisfying performance, we then design several metrics to reveal the toxic impact of inconsistent local features between visible and infrared images on the model performance. This further motivates us to develop a baseline method named Semantic Guidance Transformer (SGFormer), which utilizes high-level semantic guidance to mitigate the negative impact of local inconsistent features. Despite the simplicity of our motivation, extensive experimental results show the effectiveness of our method.

cs.CV↗