arXiv subjects

Mengqi Zhang

Publications and source records attributed to Mengqi Zhang.

At least 55 records · Page 3Linked to original sources

arXiv2024

Impact of ChatGPT on the writing style of condensed matter physicists

We apply a state-of-the-art difference-in-differences approach to estimate the impact of ChatGPT's release on the writing style of condensed matter papers on arXiv. Our analysis reveals a statistically significant improvement in the English quality of abstracts written by non-native English speakers. Importantly, this improvement remains robust even after accounting for other potential factors, confirming that it can be attributed to the release of ChatGPT. This indicates widespread adoption of the tool. Following the release of ChatGPT, there is a significant increase in the use of unique words, while the frequency of rare words decreases. Across language families, the changes in writing style are significant for authors from the Latin and Ural-Altaic groups, but not for those from the Germanic or other Indo-European groups.

cs.CL↗

arXiv2024

Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing

Large language models (LLMs) face challenges with internal knowledge inaccuracies and outdated information. Knowledge editing has emerged as a pivotal approach to mitigate these issues. Although current knowledge editing techniques exhibit promising performance in single-hop reasoning tasks, they show limitations when applied to multi-hop reasoning. Drawing on cognitive neuroscience and the operational mechanisms of LLMs, we hypothesize that the residual single-hop knowledge after editing causes edited models to revert to their original answers when processing multi-hop questions, thereby undermining their performance in multihop reasoning tasks. To validate this hypothesis, we conduct a series of experiments that empirically confirm our assumptions. Building on the validated hypothesis, we propose a novel knowledge editing method that incorporates a Knowledge Erasure mechanism for Large language model Editing (KELE). Specifically, we design an erasure function for residual knowledge and an injection function for new knowledge. Through joint optimization, we derive the optimal recall vector, which is subsequently utilized within a rank-one editing framework to update the parameters of targeted model layers. Extensive experiments on GPT-J and GPT-2 XL demonstrate that KELE substantially enhances the multi-hop reasoning capability of edited LLMs.

cs.CL↗

arXiv2024

MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model

Open-world 3D reconstruction models have recently garnered significant attention. However, without sufficient 3D inductive bias, existing methods typically entail expensive training costs and struggle to extract high-quality 3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision. Specifically, instead of using a triplane representation, we store features in 3D sparse voxels and combine transformers with 3D convolutions to leverage an explicit 3D structure and projective bias. In addition to sparse-view RGB input, we require the network to take input and generate corresponding normal maps. The input normal maps can be predicted by 2D diffusion models, significantly aiding in the guidance and refinement of the geometry's learning. Moreover, by combining Signed Distance Function (SDF) supervision with surface rendering, we directly learn to generate high-quality meshes without the need for complex multi-stage training processes. By incorporating these explicit 3D biases, MeshFormer can be trained efficiently and deliver high-quality textured meshes with fine-grained geometric details. It can also be integrated with 2D diffusion models to enable fast single-image-to-3D and text-to-3D tasks. Project page: https://meshformer3d.github.io

cs.CV↗

arXiv2024

Learning Domain-Invariant Features for Out-of-Context News Detection

Out-of-context news is a common type of misinformation on online media platforms. This involves posting a caption, alongside a mismatched news image. Existing out-of-context news detection models only consider the scenario where pre-labeled data is available for each domain, failing to address the out-of-context news detection on unlabeled domains (e.g. news topics or agencies). In this work, we therefore focus on domain adaptive out-of-context news detection. In order to effectively adapt the detection model to unlabeled news topics or agencies, we propose ConDA-TTA (Contrastive Domain Adaptation with Test-Time Adaptation) which applies contrastive learning and maximum mean discrepancy (MMD) to learn domain-invariant features. In addition, we leverage test-time target domain statistics to further assist domain adaptation. Experimental results show that our approach outperforms baselines in most domain adaptation settings on two public datasets, by as much as 2.93% in F1 and 2.08% in accuracy.

cs.CL↗

arXiv2024

Self-Supervised Position Debiasing for Large Language Models

Fine-tuning has been demonstrated to be an effective method to improve the domain performance of large language models (LLMs). However, LLMs might fit the dataset bias and shortcuts for prediction, leading to poor generation performance. Previous works have proven that LLMs are prone to exhibit position bias, i.e., leveraging information positioned at the beginning or end, or specific positional cues within the input. Existing debiasing methods for LLMs require external bias knowledge or annotated non-biased samples, which is lacking for position debiasing and impractical in reality. In this work, we propose a self-supervised position debiasing (SOD) framework to mitigate position bias for LLMs. SOD leverages unsupervised responses from pre-trained LLMs for debiasing without relying on any external knowledge. To improve the quality of unsupervised responses, we propose an objective alignment (OAM) module to prune these responses. Experiments on eight datasets and five tasks show that SOD consistently outperforms existing methods in mitigating three types of position biases. Besides, SOD achieves this by sacrificing only a small performance on biased samples, which is general and effective. To facilitate the reproducibility of the results, we share the code of all methods and datasets on https://github.com/LZKSKY/SOD.

cs.CL↗

arXiv2024

ExcluIR: Exclusionary Neural Information Retrieval

Exclusion is an important and universal linguistic skill that humans use to express what they do not want. However, in information retrieval community, there is little research on exclusionary retrieval, where users express what they do not want in their queries. In this work, we investigate the scenario of exclusionary retrieval in document retrieval for the first time. We present ExcluIR, a set of resources for exclusionary retrieval, consisting of an evaluation benchmark and a training set for helping retrieval models to comprehend exclusionary queries. The evaluation benchmark includes 3,452 high-quality exclusionary queries, each of which has been manually annotated. The training set contains 70,293 exclusionary queries, each paired with a positive document and a negative document. We conduct detailed experiments and analyses, obtaining three main observations: (1) Existing retrieval models with different architectures struggle to effectively comprehend exclusionary queries; (2) Although integrating our training data can improve the performance of retrieval models on exclusionary retrieval, there still exists a gap compared to human performance; (3) Generative retrieval models have a natural advantage in handling exclusionary queries. To facilitate future research on exclusionary retrieval, we share the benchmark and evaluation scripts on \url{https://github.com/zwh-sdu/ExcluIR}.

cs.IR↗

arXiv2024

Generative Retrieval as Multi-Vector Dense Retrieval

Generative retrieval generates identifiers of relevant documents in an end-to-end manner using a sequence-to-sequence architecture for a given query. The relation between generative retrieval and other retrieval methods, especially those based on matching within dense retrieval models, is not yet fully comprehended. Prior work has demonstrated that generative retrieval with atomic identifiers is equivalent to single-vector dense retrieval. Accordingly, generative retrieval exhibits behavior analogous to hierarchical search within a tree index in dense retrieval when using hierarchical semantic identifiers. However, prior work focuses solely on the retrieval stage without considering the deep interactions within the decoder of generative retrieval. In this paper, we fill this gap by demonstrating that generative retrieval and multi-vector dense retrieval share the same framework for measuring the relevance to a query of a document. Specifically, we examine the attention layer and prediction head of generative retrieval, revealing that generative retrieval can be understood as a special case of multi-vector dense retrieval. Both methods compute relevance as a sum of products of query and document vectors and an alignment matrix. We then explore how generative retrieval applies this framework, employing distinct strategies for computing document token vectors and the alignment matrix. We have conducted experiments to verify our conclusions and show that both paradigms exhibit commonalities of term matching in their alignment matrix.

cs.IR↗

arXiv2024

Uncovering Selective State Space Model's Capabilities in Lifelong Sequential Recommendation

Sequential Recommenders have been widely applied in various online services, aiming to model users' dynamic interests from their sequential interactions. With users increasingly engaging with online platforms, vast amounts of lifelong user behavioral sequences have been generated. However, existing sequential recommender models often struggle to handle such lifelong sequences. The primary challenges stem from computational complexity and the ability to capture long-range dependencies within the sequence. Recently, a state space model featuring a selective mechanism (i.e., Mamba) has emerged. In this work, we investigate the performance of Mamba for lifelong sequential recommendation (i.e., length>=2k). More specifically, we leverage the Mamba block to model lifelong user sequences selectively. We conduct extensive experiments to evaluate the performance of representative sequential recommendation models in the setting of lifelong sequences. Experiments on two real-world datasets demonstrate the superiority of Mamba. We found that RecMamba achieves performance comparable to the representative model while significantly reducing training duration by approximately 70% and memory costs by 80%. Codes and data are available at \url{https://github.com/nancheng58/RecMamba}.

cs.IR↗

arXiv2024

HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data

3D hand-object interaction data is scarce due to the hardware constraints in scaling up the data collection process. In this paper, we propose HOIDiffusion for generating realistic and diverse 3D hand-object interaction data. Our model is a conditional diffusion model that takes both the 3D hand-object geometric structure and text description as inputs for image synthesis. This offers a more controllable and realistic synthesis as we can specify the structure and style inputs in a disentangled manner. HOIDiffusion is trained by leveraging a diffusion model pre-trained on large-scale natural images and a few 3D human demonstrations. Beyond controllable image synthesis, we adopt the generated 3D data for learning 6D object pose estimation and show its effectiveness in improving perception systems. Project page: https://mq-zhang1.github.io/HOIDiffusion

cs.CV↗

arXiv2024

Knowledge Graph Enhanced Large Language Model Editing

Large language models (LLMs) are pivotal in advancing natural language processing (NLP) tasks, yet their efficacy is hampered by inaccuracies and outdated knowledge. Model editing emerges as a promising solution to address these challenges. However, existing editing methods struggle to track and incorporate changes in knowledge associated with edits, which limits the generalization ability of postedit LLMs in processing edited knowledge. To tackle these problems, we propose a novel model editing method that leverages knowledge graphs for enhancing LLM editing, namely GLAME. Specifically, we first utilize a knowledge graph augmentation module to uncover associated knowledge that has changed due to editing, obtaining its internal representations within LLMs. This approach allows knowledge alterations within LLMs to be reflected through an external graph structure. Subsequently, we design a graph-based knowledge edit module to integrate structured knowledge into the model editing. This ensures that the updated parameters reflect not only the modifications of the edited knowledge but also the changes in other associated knowledge resulting from the editing process. Comprehensive experiments conducted on GPT-J and GPT-2 XL demonstrate that GLAME significantly improves the generalization capabilities of post-edit LLMs in employing edited knowledge.

cs.CL↗

arXiv2023

Numerical analyses of the flow past a short rotating cylinder

This work studies the three-dimensional flow dynamics around a rotating circular cylinder of finite length, whose axis is positioned perpendicular to the streamwise direction. Direct numerical simulations and global stability analyses are performed within a parameter range of Reynolds number $Re=DU_\infty/ν<500$ (based on cylinder diameter $D$, uniform incoming flow velocity $U_\infty$), length-to-diameter ratio $AR=L/D\leq2$ and dimensionless rotation rate $α=DΩ/2U_\infty\leq2$ (where $Ω$ is rotation rate). By solving Nav\-ier--Sto\-kes equations, we investigated the wake patterns and explored the phase diagrams of the lift and drag coefficients. For a cylinder with $AR=1$, we found that when the rotation effect is weak ($0\leqα\lesssim0.3$), the wake pattern is similar to the unsteady wake past the non-rotating finite-length cylinder, but with a new linear unstable mode competing to dominate the saturation state of the wake. The flow becomes stable for $0.3\lesssimα\lesssim0.9$ when $Re<360$. When the rotation effect is strong ($α\gtrsim0.9$), new low-frequency wake patterns with stronger oscillations emerge. Furthermore, the stability analyses based on the time-averaged flows and on the steady solutions demonstrate the existence of multiple unstable modes undergoing Hopf bifurcation, greatly influenced by the rotation effect. The shapes of these global eigenmodes are presented and compared, as well as their structural sensitivity, visualising the flow region important for the disturbance development with rotation. This research contributes to our understanding of the complex bluff-body wake dynamics past this critical configuration.

physics.flu-dyn↗

arXiv2023

Patched Denoising Diffusion Models For High-Resolution Image Synthesis

We propose an effective denoising diffusion model for generating high-resolution images (e.g., 1024$\times$512), trained on small-size image patches (e.g., 64$\times$64). We name our algorithm Patch-DM, in which a new feature collage strategy is designed to avoid the boundary artifact when synthesizing large-size images. Feature collage systematically crops and combines partial features of the neighboring patches to predict the features of a shifted image patch, allowing the seamless generation of the entire image due to the overlap in the patch feature space. Patch-DM produces high-quality image synthesis results on our newly collected dataset of nature images (1024$\times$512), as well as on standard benchmarks of smaller sizes (256$\times$256), including LSUN-Bedroom, LSUN-Church, and FFHQ. We compare our method with previous patch-based generation methods and achieve state-of-the-art FID scores on all four datasets. Further, Patch-DM also reduces memory complexity compared to the classic diffusion models.

cs.CV↗

arXiv2023

Numerical analyses of wire-plate electrohydrodynamic flows

We present numerical analyses of 2-D electrohydrodynamic (EHD) flows of a dielectric liquid between a wire electrode and two plate electrodes with a Poiseuille flow, using direct numerical simulation and global stability analysis. Both conduction and injection mechanisms for charge generation are considered. In this work, we focused on the intensity of the cross-flow and studied the EHD flows without a cross-flow, with a weak cross-flow and with a strong cross-flow. (1)In the case without a cross-flow, we investigated its nonlinear flow structures and linear dynamics. We found that the flow in the conduction regime is steady, whereas the flow in the injection regime is oscillatory, which can be explained by a global stability analysis. (2)The EHD flow with a weak cross-flow is closely related to the flow phenomena in electrostatic precipitator (ESP). Our analyses indicate that increasing the cross-flow intensity or the electric Reynolds number leads to a less stable flow. Based on these results, we infer that one should adopt a relatively low voltage and weak cross-flow in the wire-plate EHD flow to avoid flow instability, which may hold practical implications for ESP. (3)The case of strong cross-flow is examined to study the EHD effect on the wake flow. By comparing the conventional cylindrical wake with the EHD wake in linear and nonlinear regimes, we found that the EHD effect brings forward the vortex shedding in wake flows. Besides, the EHD effect reduces the drag coefficient when the cross-flow is weak, but increases it when it is strong.

physics.flu-dyn↗

arXiv2023

MetaTKG: Learning Evolutionary Meta-Knowledge for Temporal Knowledge Graph Reasoning

Reasoning over Temporal Knowledge Graphs (TKGs) aims to predict future facts based on given history. One of the key challenges for prediction is to learn the evolution of facts. Most existing works focus on exploring evolutionary information in history to obtain effective temporal embeddings for entities and relations, but they ignore the variation in evolution patterns of facts, which makes them struggle to adapt to future data with different evolution patterns. Moreover, new entities continue to emerge along with the evolution of facts over time. Since existing models highly rely on historical information to learn embeddings for entities, they perform poorly on such entities with little historical information. To tackle these issues, we propose a novel Temporal Meta-learning framework for TKG reasoning, MetaTKG for brevity. Specifically, our method regards TKG prediction as many temporal meta-tasks, and utilizes the designed Temporal Meta-learner to learn evolutionary meta-knowledge from these meta-tasks. The proposed method aims to guide the backbones to learn to adapt quickly to future data and deal with entities with little historical information by the learned meta-knowledge. Specially, in temporal meta-learner, we design a Gating Integration module to adaptively establish temporal correlations between meta-tasks. Extensive experiments on four widely-used datasets and three backbones demonstrate that our method can greatly improve the performance.

cs.AI↗

arXiv2022

Reinforcement-learning-based control of convectively-unstable flows

This work reports the application of a model-free deep-reinforcement-learning-based (DRL) flow control strategy to suppress perturbations evolving in the 1-D linearised Kuramoto-Sivashinsky (KS) equation and 2-D boundary layer flows. The former is commonly used to model the disturbance developing in flat-plate boundary layer flows. These flow systems are convectively unstable, being able to amplify the upstream disturbance, and are thus difficult to control. The control action is implemented through a volumetric force at a fixed position and the control performance is evaluated by the reduction of perturbation amplitude downstream. We first demonstrate the effectiveness of the DRL-based control in the KS system subjected to a random upstream noise. The amplitude of perturbation monitored downstream is significantly reduced and the learnt policy is shown to be robust to both measurement and external noise. One of our focuses is to optimally place sensors in the DRL control using the gradient-free particle swarm optimisation algorithm. After the optimisation process for different numbers of sensors, a specific eight-sensor placement is found to yield the best control performance. The optimised sensor placement in the KS equation is applied directly to control 2-D Blasius boundary layer flows and can efficiently reduce the downstream perturbation energy. Via flow analyses, the control mechanism found by DRL is the opposition control. Besides, it is found that when the flow instability information is embedded in the reward function of DRL to penalise the instability, the control performance can be further improved in this convectively-unstable flow.

physics.flu-dyn↗

arXiv2022

Moffatt eddies in electrohydrodynamic flows: numerical simulations and analyses

We study numerically a sequence of eddies in two-dimensional electrohydrodynamic (EHD) flows of a dielectric liquid, driven by an electric potential difference between a hyperbolic blade electrode and a flat plate electrode (or the blade-plate configuration). The electrically-driven flow impinges on the plate to generate vortices, which resemble Moffatt eddies (Moffatt, $\textit{J. Fluid Mech.}$ vol. 18, 1-18, 1964). Such a phenomenon in EHD was first reported in the experimental work of Perri $\textit{et al., J. Fluid Mech.}$ vol. 900, A12, 2020. We conduct direct numerical simulations of the EHD flow with three Moffatt-type eddies in a large computational domain at moderate electric Rayleigh numbers ($ T $, quantifying the strength of the electric field). The ratios of size and intensity of the adjacent eddies are examined and they can be compared favourably to the theoretical prediction of Moffatt; interestingly, the quantitative comparison is remarkably accurate for the two eddies in the farfield. Our investigation also shows that a larger $T$ strengthens the vortex intensity, and a stronger charge diffusion effect enlarges the vortex size. A sufficiently large $T$ can further result in an oscillating flow, consistent with the experimental observation. In addition, a global stability analysis of the steady blade-plate EHD flow is conducted. The global mode is detailedly characterised at different values of $T$. When $T$ is large, the confinement effect of the geometry in the center region may lead to an increased oscillation frequency. This work contributes to the quantitative characterisation of the Moffatt-type eddies in electrohydrodynamic flows.

physics.flu-dyn↗

arXiv2022

FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration

To accomplish punctuation restoration, most existing methods focus on introducing extra information (e.g., part-of-speech) or addressing the class imbalance problem. Recently, large-scale transformer-based pre-trained language models (PLMS) have been utilized widely and obtained remarkable success. However, the PLMS are trained on the large dataset with marks, which may not fit well with the small dataset without marks, causing the convergence to be not ideal. In this study, we propose a Feature Fusion two-stream framework (FF2) to bridge the gap. Specifically, one stream leverages a pre-trained language model to capture the semantic feature, while another auxiliary module captures the feature at hand. We also modify the computation of multi-head attention to encourage communication among heads. Then, two features with different perspectives are aggregated to fuse information and enhance context awareness. Without additional data, the experimental results on the popular benchmark IWSLT demonstrate that FF2 achieves new SOTA performance, which verifies that our approach is effective.

cs.CL↗

arXiv2022

Numerical Study of Viscoelastic Upstream Instability

In this work, we report numerical results on the flow instability and bifurcation of a viscoelastic fluid in the upstream region of a confined cylinder in a narrow channel. Two-dimensional direct numerical simulations based on the FENE-P model (the finite-extensible nonlinear elastic model with the Peterlin closure) are conducted with numerical stabilization techniques. Our results show that the macroscopic viscoelastic constitutive relation can capture the viscoelastic upstream instability reported in previous experiments for low-Reynolds-number flows. The numerical simulations reveal that the non-dimensional recirculation length ($L_D$) is affected by the cylinder blocking rate ($BR$), the Weissenberg number ($Wi$), the viscosity ratio ($β$), and the maximum polymer extension ($L$). Close to the onset of upstream recirculation, depending on the values of $L$ and $β$, the bifurcation can be supercritical or subcritical. $β$ has a nonlinear influence on the upstream recirculation length. This work contributes to our theoretical understanding of this new instability mechanism in viscoelastic wake flows.

physics.flu-dyn↗