arXiv subjects

Jie Zhang

Publications and source records attributed to Jie Zhang.

At least 541 records · Page 30Linked to original sources

arXiv2024

LCM-SVC: Latent Diffusion Model Based Singing Voice Conversion with Inference Acceleration via Latent Consistency Distillation

Any-to-any singing voice conversion (SVC) aims to transfer a target singer's timbre to other songs using a short voice sample. However many diffusion model based any-to-any SVC methods, which have achieved impressive results, usually suffered from low efficiency caused by a mass of inference steps. In this paper, we propose LCM-SVC, a latent consistency distillation (LCD) based latent diffusion model (LDM) to accelerate inference speed. We achieved one-step or few-step inference while maintaining the high performance by distilling a pre-trained LDM based SVC model, which had the advantages of timbre decoupling and sound quality. Experimental results show that our proposed method can significantly reduce the inference time and largely preserve the sound quality and timbre similarity comparing with other state-of-the-art SVC models. Audio samples are available at https://sounddemos.github.io/lcm-svc.

eess.AS↗

arXiv2024

Zeeman field induced corner states in the Kane-Mele-Hubbard model

We investigate how the in-plane Zeeman field and the Hubbard interaction can jointly affect the topological states in the Kane-Mele-Hubbard model. At low Zeeman field, the projector quantum Monte Carlo (PQMC) simulations demonstrate Mott transitions between the higher-order topological insulator induced by the in-plane Zeeman field and the antiferromagnetic insulator induced by the Hubbard interaction. The interacting higher-order topological state on a finite-sized honeycomb lattice is characterized by the local density of states derived from the real-space spectral function. In the higher-order topological phase, the mirror-inversion symmetry protected corner states exist on the diamond-shaped honeycomb lattice. By comparing the spatial distribution of corner states between the Kane-Mele and Kane-Mele-Hubbard models, we show that the effect of the Hubbard interaction is to contribute some extra in-plane Zeeman field. At the mean-field level, we calculate the full phase diagram of the Kane-Mele-Hubbard model under the influence of the in-plane Zeeman field. The mean-field phase diagram shows that a high in-plane Zeeman field drives the system into a spin-polarized (topologically trivial) phase. In particular, the upper limit of the Zeeman field that can induce corner states in the Kane-Mele model is found to be $h_c = 1.0(3)$. In the parameter region of PQMC simulations, the mean-field solutions are in fair agreement with the PQMC results.

cond-mat.str-el↗

arXiv2024

Report on the Advanced Linear Collider Study Group (ALEGRO) Workshop 2024

The workshop focused on the application of ANAs to particle physics keeping in mind the ultimate goal of a collider at the energy frontier (10\,TeV, e$^+$/e$^-$, e$^-$/e$^-$, or $γγ$). The development of ANAs is conducted at universities and national laboratories worldwide. The community is thematically broad and diverse, in particular since lasers suitable for ANA research (multi-hundred-terawatt peak power, a few tens of femtosecond-long pulses) and acceleration of electrons to hundreds of mega electron volts to multi giga electron volts became commercially available. The community spans several continents (Europe, America, Asia), including more than 62 laboratories in more than 20 countries. It is among the missions of the ICFA-ANA panel to feature the amazing progress made with ANAs, to provide international coordination and to foster international collaborations towards a future HEP collider. The scope of this edition of the workshop was to discuss the recent progress and necessary steps towards realizing a linear collider for particle physics based on novel-accelerator technologies (laser or beam driven in plasma or structures). Updates on the relevant aspects of the European Strategy for Particle Physics (ESPP) Roadmap Process as well as of the P5 (in the US) were presented, and ample time was dedicated to discussions. The major outcome of the workshop is the decision for ALEGRO to coordinate efforts in Europe, in the US, and in Asia towards a pre-CDR for an ANA-based, 10\,TeV CM collider. This goal of this coordination is to lead to a funding proposal to be submitted to both EU and EU/US funding agencies. This document presents a summary of the workshop, as seen by the co-chairs, as well as short 'one-pagers' written by the presenters at the workshop.

physics.acc-ph↗

arXiv2024

FacialPulse: An Efficient RNN-based Depression Detection via Temporal Facial Landmarks

Depression is a prevalent mental health disorder that significantly impacts individuals' lives and well-being. Early detection and intervention are crucial for effective treatment and management of depression. Recently, there are many end-to-end deep learning methods leveraging the facial expression features for automatic depression detection. However, most current methods overlook the temporal dynamics of facial expressions. Although very recent 3DCNN methods remedy this gap, they introduce more computational cost due to the selection of CNN-based backbones and redundant facial features. To address the above limitations, by considering the timing correlation of facial expressions, we propose a novel framework called FacialPulse, which recognizes depression with high accuracy and speed. By harnessing the bidirectional nature and proficiently addressing long-term dependencies, the Facial Motion Modeling Module (FMMM) is designed in FacialPulse to fully capture temporal features. Since the proposed FMMM has parallel processing capabilities and has the gate mechanism to mitigate gradient vanishing, this module can also significantly boost the training speed. Besides, to effectively use facial landmarks to replace original images to decrease information redundancy, a Facial Landmark Calibration Module (FLCM) is designed to eliminate facial landmark errors to further improve recognition accuracy. Extensive experiments on the AVEC2014 dataset and MMDA dataset (a depression dataset) demonstrate the superiority of FacialPulse on recognition accuracy and speed, with the average MAE (Mean Absolute Error) decreased by 21% compared to baselines, and the recognition speed increased by 100% compared to state-of-the-art methods. Codes are released at https://github.com/volatileee/FacialPulse.

cs.CV↗

arXiv2024

The Crowd in MOOCs: A Study of Learning Patterns at Scale

The increasing availability of learning activity data in Massive Open Online Courses (MOOCs) enables us to conduct a large-scale analysis of learners' learning behavior. In this paper, we analyze a dataset of 351 million learning activities from 0.8 million unique learners enrolled in over 1.6 thousand courses within two years. Specifically, we mine and identify the learning patterns of the crowd from both temporal and course enrollment perspectives leveraging mutual information theory and sequential pattern mining methods. From the temporal perspective, we find that the time intervals between consecutive learning activities of learners exhibit a mix of power-law and periodic cosine function distribution. By qualifying the relationship between course pairs, we observe that the most frequently co-enrolled courses usually fall in the same category or the same university. We demonstrate these findings can facilitate manifold applications including recommendation tasks on courses. A simple recommendation model utilizing the course enrollment patterns is competitive to the baselines with 200$\times$ faster training time.

cs.IR↗

arXiv2024

Leveraging Knowledge Graph Embedding for Effective Conversational Recommendation

Conversational recommender system (CRS), which combines the techniques of dialogue system and recommender system, has obtained increasing interest recently. In contrast to traditional recommender system, it learns the user preference better through interactions (i.e. conversations), and then further boosts the recommendation performance. However, existing studies on CRS ignore to address the relationship among attributes, users, and items effectively, which might lead to inappropriate questions and inaccurate recommendations. In this view, we propose a knowledge graph based conversational recommender system (referred as KG-CRS). Specifically, we first integrate the user-item graph and item-attribute graph into a dynamic graph, i.e., dynamically changing during the dialogue process by removing negative items or attributes. We then learn informative embedding of users, items, and attributes by also considering propagation through neighbors on the graph. Extensive experiments on three real datasets validate the superiority of our method over the state-of-the-art approaches in terms of both the recommendation and conversation tasks.

cs.IR↗

arXiv2024

Mechanism Design for Locating Facilities with Capacities with Insufficient Resources

This paper explores the Mechanism Design aspects of the $m$-Capacitated Facility Location Problem where the total facility capacity is less than the number of agents. Following the framework outlined by Aziz et al., the Social Welfare of the facility location is determined through a First-Come-First-Served (FCFS) game, in which agents compete once the facility positions are established. When the number of facilities is $m > 1$, the Nash Equilibrium (NE) of the FCFS game is not unique, making the utility of the agents and the concept of truthfulness unclear. To tackle these issues, we consider absolutely truthful mechanisms, i.e. mechanisms that prevent agents from misreporting regardless of the strategies used during the FCFS game. We combine this stricter truthfulness requirement with the notion of Equilibrium Stable (ES) mechanisms, which are mechanisms whose Social Welfare does not depend on the NE of the FCFS game. We demonstrate that the class of percentile mechanisms is absolutely truthful and identify the conditions under which they are ES. We also show that the approximation ratio of each ES percentile mechanism is bounded and determine its value. Notably, when all the facilities have the same capacity and the number of agents is sufficiently large, it is possible to achieve an approximation ratio smaller than $1+\frac{1}{2m-1}$. Finally, we extend our study to encompass higher-dimensional problems. Within this framework, we demonstrate that the class of ES percentile mechanisms is even more restricted and characterize the mechanisms that are both ES and absolutely truthful. We further support our findings by empirically evaluating the performance of the mechanisms when the agents are the samples of a distribution.

cs.GT↗

arXiv2024

A generalization of the Hamiltonian cycle in dense digraphs

Let D be a digraph and C be a cycle in D. For any two vertices x and y in D, the distance from x to y is the minimum length of a path from x to y. We denote the square of Let $D$ be a digraph and $C$ be a cycle in $D$. For any two vertices $x$ and $y$ in $D$, the distance from $x$ to $y$ is the minimum length of a path from $x$ to $y$. We denote the square of the cycle $C$ to be the graph whose vertex set is $V(C)$ and for distinct vertices $x$ and $y$ in $C$, there is an arc from $x$ to $y$ if and only if the distance from $x$ to $y$ in $C$ is at most $2$. The reverse square of the cycle $C$ is the digraph with the same vertex set as $C$, and the arc set $A(C)\cup \{yx: \mbox{the vertices}\ x, y\in V(C)\ \mbox{and the distance from $x$ to $y$ on $C$ is $2$}\}$. In this paper, we show that for any real number $γ>0$ there exists a constant $n_0=n_0(γ)$, such that every digraph on $n\geq n_0$ vertices with the minimum in- and out-degree at least $(2/3+γ)n$ contains the reverse square of a Hamiltonian cycle. Our result extends a result of Czygrinow, Kierstead and Molla.

math.CO↗

arXiv2024

Turning Your Strength into Watermark: Watermarking Large Language Model via Knowledge Injection

Large language models (LLMs) have demonstrated outstanding performance, making them valuable digital assets with significant commercial potential. Unfortunately, the LLM and its API are susceptible to intellectual property theft. Watermarking is a classic solution for copyright verification. However, most recent emerging LLM watermarking methods focus on identifying AI-generated texts rather than watermarking LLM itself. Only a few attempts are based on weight quantification and backdoor watermarking, which are not robust or covert enough, limiting their applicability in practice. To address this issue, we propose a novel watermarking method for LLMs based on knowledge injection and innovatively use knowledge as the watermark carrier. Specifically, in the watermark embedding stage, we first embed the watermarks into the selected knowledge to obtain the watermarked knowledge, subsequently injected into the to-be-protected LLM. In the watermark extraction stage, questions related to the watermarked knowledge are designed, for querying the suspect LLM and extracting the watermarks from its response. The experiments show that the watermark extraction success rate is close to 100% and demonstrate the effectiveness, fidelity, stealthiness, and robustness of our proposed method.

cs.CR↗

arXiv2024

Sampling to Distill: Knowledge Transfer from Open-World Data

Data-Free Knowledge Distillation (DFKD) is a novel task that aims to train high-performance student models using only the pre-trained teacher network without original training data. Most of the existing DFKD methods rely heavily on additional generation modules to synthesize the substitution data resulting in high computational costs and ignoring the massive amounts of easily accessible, low-cost, unlabeled open-world data. Meanwhile, existing methods ignore the domain shift issue between the substitution data and the original data, resulting in knowledge from teachers not always trustworthy and structured knowledge from data becoming a crucial supplement. To tackle the issue, we propose a novel Open-world Data Sampling Distillation (ODSD) method for the DFKD task without the redundant generation process. First, we try to sample open-world data close to the original data's distribution by an adaptive sampling module and introduce a low-noise representation to alleviate the domain shift issue. Then, we build structured relationships of multiple data examples to exploit data knowledge through the student model itself and the teacher's structured representation. Extensive experiments on CIFAR-10, CIFAR-100, NYUv2, and ImageNet show that our ODSD method achieves state-of-the-art performance with lower FLOPs and parameters. Especially, we improve 1.50\%-9.59\% accuracy on the ImageNet dataset and avoid training the separate generator for each class.

cs.CV↗

arXiv2024

DiffClass: Diffusion-Based Class Incremental Learning

Class Incremental Learning (CIL) is challenging due to catastrophic forgetting. On top of that, Exemplar-free Class Incremental Learning is even more challenging due to forbidden access to previous task data. Recent exemplar-free CIL methods attempt to mitigate catastrophic forgetting by synthesizing previous task data. However, they fail to overcome the catastrophic forgetting due to the inability to deal with the significant domain gap between real and synthetic data. To overcome these issues, we propose a novel exemplar-free CIL method. Our method adopts multi-distribution matching (MDM) diffusion models to unify quality and bridge domain gaps among all domains of training data. Moreover, our approach integrates selective synthetic image augmentation (SSIA) to expand the distribution of the training data, thereby improving the model's plasticity and reinforcing the performance of our method's ultimate component, multi-domain adaptation (MDA). With the proposed integrations, our method then reformulates exemplar-free CIL into a multi-domain adaptation problem to implicitly address the domain gap problem to enhance model stability during incremental training. Extensive experiments on benchmark class incremental datasets and settings demonstrate that our method excels previous exemplar-free CIL methods and achieves state-of-the-art performance.

cs.CV↗

arXiv2024

Training-Free Large Model Priors for Multiple-in-One Image Restoration

Image restoration aims to reconstruct the latent clear images from their degraded versions. Despite the notable achievement, existing methods predominantly focus on handling specific degradation types and thus require specialized models, impeding real-world applications in dynamic degradation scenarios. To address this issue, we propose Large Model Driven Image Restoration framework (LMDIR), a novel multiple-in-one image restoration paradigm that leverages the generic priors from large multi-modal language models (MMLMs) and the pretrained diffusion models. In detail, LMDIR integrates three key prior knowledges: 1) global degradation knowledge from MMLMs, 2) scene-aware contextual descriptions generated by MMLMs, and 3) fine-grained high-quality reference images synthesized by diffusion models guided by MMLM descriptions. Standing on above priors, our architecture comprises a query-based prompt encoder, degradation-aware transformer block injecting global degradation knowledge, content-aware transformer block incorporating scene description, and reference-based transformer block incorporating fine-grained image priors. This design facilitates single-stage training paradigm to address various degradations while supporting both automatic and user-guided restoration. Extensive experiments demonstrate that our designed method outperforms state-of-the-art competitors on multiple evaluation benchmarks.

cs.CV↗

arXiv2024

Robust-Wide: Robust Watermarking against Instruction-driven Image Editing

Instruction-driven image editing allows users to quickly edit an image according to text instructions in a forward pass. Nevertheless, malicious users can easily exploit this technique to create fake images, which could cause a crisis of trust and harm the rights of the original image owners. Watermarking is a common solution to trace such malicious behavior. Unfortunately, instruction-driven image editing can significantly change the watermarked image at the semantic level, making current state-of-the-art watermarking methods ineffective. To remedy it, we propose Robust-Wide, the first robust watermarking methodology against instruction-driven image editing. Specifically, we follow the classic structure of deep robust watermarking, consisting of the encoder, noise layer, and decoder. To achieve robustness against semantic distortions, we introduce a novel Partial Instruction-driven Denoising Sampling Guidance (PIDSG) module, which consists of a large variety of instruction injections and substantial modifications of images at different semantic levels. With PIDSG, the encoder tends to embed the watermark into more robust and semantic-aware areas, which remains in existence even after severe image editing. Experiments demonstrate that Robust-Wide can effectively extract the watermark from the edited image with a low bit error rate of nearly 2.6% for 64-bit watermark messages. Meanwhile, it only induces a neglectable influence on the visual quality and editability of the original images. Moreover, Robust-Wide holds general robustness against different sampling configurations and other popular image editing methods such as ControlNet-InstructPix2Pix, MagicBrush, Inpainting, and DDIM Inversion. Codes and models are available at https://github.com/hurunyi/Robust-Wide.

cs.CR↗

arXiv2024

T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models

While text-to-image diffusion models demonstrate impressive generation capabilities, they also exhibit vulnerability to backdoor attacks, which involve the manipulation of model outputs through malicious triggers. In this paper, for the first time, we propose a comprehensive defense method named T2IShield to detect, localize, and mitigate such attacks. Specifically, we find the "Assimilation Phenomenon" on the cross-attention maps caused by the backdoor trigger. Based on this key insight, we propose two effective backdoor detection methods: Frobenius Norm Threshold Truncation and Covariance Discriminant Analysis. Besides, we introduce a binary-search approach to localize the trigger within a backdoor sample and assess the efficacy of existing concept editing methods in mitigating backdoor attacks. Empirical evaluations on two advanced backdoor attack scenarios show the effectiveness of our proposed defense method. For backdoor sample detection, T2IShield achieves a detection F1 score of 88.9$\%$ with low computational cost. Furthermore, T2IShield achieves a localization F1 score of 86.4$\%$ and invalidates 99$\%$ poisoned samples. Codes are released at https://github.com/Robin-WZQ/T2IShield.

cs.CV↗

arXiv2024

The Better Angels of Machine Personality: How Personality Relates to LLM Safety

Personality psychologists have analyzed the relationship between personality and safety behaviors in human society. Although Large Language Models (LLMs) demonstrate personality traits, the relationship between personality traits and safety abilities in LLMs still remains a mystery. In this paper, we discover that LLMs' personality traits are closely related to their safety abilities, i.e., toxicity, privacy, and fairness, based on the reliable MBTI-M scale. Meanwhile, the safety alignment generally increases various LLMs' Extraversion, Sensing, and Judging traits. According to such findings, we can edit LLMs' personality traits and improve their safety performance, e.g., inducing personality from ISTJ to ISTP resulted in a relative improvement of approximately 43% and 10% in privacy and fairness performance, respectively. Additionally, we find that LLMs with different personality traits are differentially susceptible to jailbreak. This study pioneers the investigation of LLM safety from a personality perspective, providing new insights into LLM safety enhancement.

cs.CL↗

arXiv2024

Flocking by Turning Away

Flocking, as paradigmatically exemplified by birds, is the coherent collective motion of active agents. As originally conceived, flocking emerges through alignment interactions between the agents. Here, we report that flocking can also emerge through interactions that turn agents away from each other. Combining simulations, kinetic theory, and experiments, we demonstrate this mechanism of flocking in self-propelled Janus colloids with stronger repulsion on the front than on the rear. The polar state is stable because particles achieve a compromise between turning away from left and right neighbors. Unlike for alignment interactions, the emergence of polar order from turn-away interactions requires particle repulsion. At high concentration, repulsion produces flocking Wigner crystals. Whereas repulsion often leads to motility-induced phase separation of active particles, here it combines with turn-away torques to produce flocking. Therefore, our findings bridge the classes of aligning and non-aligning active matter. Our results could help to reconcile the observations that cells can flock despite turning away from each other via contact inhibition of locomotion. Overall, our work shows that flocking is a very robust phenomenon that arises even when the orientational interactions would seem to prevent it.

cond-mat.soft↗

arXiv2024

RB-SQL: A Retrieval-based LLM Framework for Text-to-SQL

Large language models (LLMs) with in-context learning have significantly improved the performance of text-to-SQL task. Previous works generally focus on using exclusive SQL generation prompt to improve the LLMs' reasoning ability. However, they are mostly hard to handle large databases with numerous tables and columns, and usually ignore the significance of pre-processing database and extracting valuable information for more efficient prompt engineering. Based on above analysis, we propose RB-SQL, a novel retrieval-based LLM framework for in-context prompt engineering, which consists of three modules that retrieve concise tables and columns as schema, and targeted examples for in-context learning. Experiment results demonstrate that our model achieves better performance than several competitive baselines on public datasets BIRD and Spider.

cs.CL↗

arXiv2024

The k-Facility Location Problem Via Optimal Transport: A Bayesian Study of the Percentile Mechanisms

In this paper, we investigate the $k$-Facility Location Problem ($k$-FLP) within the Bayesian Mechanism Design framework, in which agents' preferences are samples of a probability distributed on a line. Our primary contribution is characterising the asymptotic behavior of percentile mechanisms, which varies according to the distribution governing the agents' types. To achieve this, we connect the $k$-FLP and projection problems in the Wasserstein space. Owing to this relation, we show that the ratio between the expected cost of a percentile mechanism and the expected optimal cost is asymptotically bounded. Furthermore, we characterize the limit of this ratio and analyze its convergence speed. Our asymptotic study is complemented by deriving an upper bound on the Bayesian approximation ratio, applicable when the number of agents $n$ exceeds the number of facilities $k$. We also characterize the optimal percentile mechanism for a given agent's distribution through a system of $k$ equations. Finally, we estimate the optimality loss incurred when the optimal percentile mechanism is derived using an approximation of the agents' distribution rather than the actual distribution.

cs.GT↗