arXiv subjects

Lei Meng

Publications and source records attributed to Lei Meng.

At least 55 records · Page 3Linked to original sources

arXiv2024

Plug-in Diffusion Model for Sequential Recommendation

Pioneering efforts have verified the effectiveness of the diffusion models in exploring the informative uncertainty for recommendation. Considering the difference between recommendation and image synthesis tasks, existing methods have undertaken tailored refinements to the diffusion and reverse process. However, these approaches typically use the highest-score item in corpus for user interest prediction, leading to the ignorance of the user's generalized preference contained within other items, thereby remaining constrained by the data sparsity issue. To address this issue, this paper presents a novel Plug-in Diffusion Model for Recommendation (PDRec) framework, which employs the diffusion model as a flexible plugin to jointly take full advantage of the diffusion-generating user preferences on all items. Specifically, PDRec first infers the users' dynamic preferences on all items via a time-interval diffusion model and proposes a Historical Behavior Reweighting (HBR) mechanism to identify the high-quality behaviors and suppress noisy behaviors. In addition to the observed items, PDRec proposes a Diffusion-based Positive Augmentation (DPA) strategy to leverage the top-ranked unobserved items as the potential positive samples, bringing in informative and diverse soft signals to alleviate data sparsity. To alleviate the false negative sampling issue, PDRec employs Noise-free Negative Sampling (NNS) to select stable negative samples for ensuring effective model optimization. Extensive experiments and analyses on four datasets have verified the superiority of the proposed PDRec over the state-of-the-art baselines and showcased the universality of PDRec as a flexible plugin for commonly-used sequential encoders in different recommendation scenarios. The code is available in https://github.com/hulkima/PDRec.

cs.IR↗

arXiv2023

RewriteLM: An Instruction-Tuned Large Language Model for Text Rewriting

Large Language Models (LLMs) have demonstrated impressive capabilities in creative tasks such as storytelling and E-mail generation. However, as LLMs are primarily trained on final text results rather than intermediate revisions, it might be challenging for them to perform text rewriting tasks. Most studies in the rewriting tasks focus on a particular transformation type within the boundaries of single sentences. In this work, we develop new strategies for instruction tuning and reinforcement learning to better align LLMs for cross-sentence rewriting tasks using diverse wording and structures expressed through natural languages including 1) generating rewriting instruction data from Wiki edits and public corpus through instruction generation and chain-of-thought prompting; 2) collecting comparison data for reward model training through a new ranking function. To facilitate this research, we introduce OpenRewriteEval, a novel benchmark covers a wide variety of rewriting types expressed through natural language instructions. Our results show significant improvements over a variety of baselines. The public repository is available on GitHub under Google Research (https://github.com/google-research/google-research/tree/master/rewritelm).

cs.CL↗

arXiv2023

SiRA: Sparse Mixture of Low Rank Adaptation

Parameter Efficient Tuning has been an prominent approach to adapt the Large Language Model to downstream tasks. Most previous works considers adding the dense trainable parameters, where all parameters are used to adapt certain task. We found this less effective empirically using the example of LoRA that introducing more trainable parameters does not help. Motivated by this we investigate the importance of leveraging "sparse" computation and propose SiRA: sparse mixture of low rank adaption. SiRA leverages the Sparse Mixture of Expert(SMoE) to boost the performance of LoRA. Specifically it enforces the top $k$ experts routing with a capacity limit restricting the maximum number of tokens each expert can process. We propose a novel and simple expert dropout on top of gating network to reduce the over-fitting issue. Through extensive experiments, we verify SiRA performs better than LoRA and other mixture of expert approaches across different single tasks and multitask settings.

cs.CL↗

arXiv2023

Impact of recent measurement of $(g-2)_μ$, LHC search for supersymmetry, and LZ experiment on Minimal Supersymmetric Standard Model

Motivated by the recent measurement of muon anomalous magnetic moment at Fermilab, the rapid progress of the LHC search for supersymmetry, and the significantly improved sensitivities of dark matter direct detection experiments, we studied their impacts on the Minimal Supersymmetric Standard Model (MSSM). We conclude that higgsino mass should be larger than about $500~{\rm GeV}$ for $M_1 < 0 $ and $630~{\rm GeV}$ for $M_1 > 100~{\rm GeV}$, where $M_1$ denotes the bino mass. These improved bounds imply a tuning of ${\cal{O}}(1\%)$ to predict the $Z$-boson mass and simultaneously worsen the naturalness of the $Z$- and $h$-mediated resonant annihilations to achieve the measured dark matter density. We also conclude that the LHC restrictions have set lower bounds on the sparticle mass spectra: $ m_{\tildeχ_1^0} \gtrsim 210~{\rm GeV}$, $m_{\tildeχ_2^0}, m_{\tildeχ_1^\pm} \gtrsim 235~{\rm GeV}$, $m_{\tildeχ_3^0} \gtrsim 515~{\rm GeV}$, $m_{\tildeχ_4^0} \gtrsim 525~{\rm GeV}$, $m_{\tildeχ_2^\pm} \gtrsim 530~{\rm GeV}$, $m_{\tildeν_μ} \gtrsim 235~{\rm GeV}$, $ m_{\tildeμ_1} \gtrsim 215~{\rm GeV}$, and $m_{\tildeμ_2} \gtrsim 250~{\rm GeV}$, where $\tildeχ_{2}^0$ and $\tildeχ_1^\pm$ are wino-dominated when they are lighter than about $500~{\rm GeV}$. These bounds are far beyond the reach of the LEP experiments in searching for supersymmetry and have not been acquired before. In addition, we illuminate how some parameter spaces of the MSSM have been tested at the LHC and provide five scenarios in which the theory coincides with the LHC restrictions. Once the muon g-2 anomaly is confirmed to originate from supersymmetry, this research may serve as a guide to explore the characteristics of the MSSM in future experiments.

hep-ph↗

arXiv2023

95 GeV Diphoton and $b \bar{b}$ Excesses in the General Next-to-Minimal Supersymmetric Standard Model

The CMS and ATLAS collaborations recently published their results searching for light Higgs bosons, using the complete Run 2 data of the LHC. Both reported an excess in the diphoton invariant mass distribution at $m_{γγ} \simeq 95.4~{\rm GeV}$ with compatible signal strengths. The combined result corresponded to a local significance of $3.1σ$. Besides, the mass of the diphoton signal coincided with that of the $b\bar{b}$ excess observed at the LEP. Given the remarkable theoretical advantages of the general Next-to-Minimal Supersymmetric Standard Model, we interpret these excesses by the resonant productions of the singlet-dominated CP-even Higgs boson predicted by the theory. Using both analytic formulae and numerical results, we show that the idea can interpret the excesses by broad parameter space without contradicting current experimental restrictions, including those from the 125~{\rm GeV} Higgs data, the dark matter relic abundance and direct detection experiments, and the collider searches for supersymmetry and extra Higgs bosons. Although the explanations are scarcely affected by present Higgs data and the LHC search for supersymmetry, the dark matter physics may leave footprints on them. We also survey the other signals of the light Higgs boson at the LHC.

hep-ph↗

arXiv2023

Critique Ability of Large Language Models

Critical thinking is essential for rational decision-making and problem-solving. This skill hinges on the ability to provide precise and reasoned critiques and is a hallmark of human intelligence. In the era of large language models (LLMs), this study explores the ability of LLMs to deliver accurate critiques across various tasks. We are interested in this topic as a capable critic model could not only serve as a reliable evaluator, but also as a source of supervised signals for model tuning. Particularly, if a model can self-critique, it has the potential for autonomous self-improvement. To examine this, we introduce a unified evaluation framework for assessing the critique abilities of LLMs. We develop a benchmark called CriticBench, which comprises 3K high-quality natural language queries and corresponding model responses; and annotate the correctness of these responses. The benchmark cover tasks such as math problem-solving, code completion, and question answering. We evaluate multiple LLMs on the collected dataset and our analysis reveals several noteworthy insights: (1) Critique is generally challenging for most LLMs, and this capability often emerges only when models are sufficiently large. (2) In particular, self-critique is especially difficult. Even top-performing LLMs struggle to achieve satisfactory performance. (3) Models tend to have lower critique accuracy on problems where they are most uncertain. To this end, we introduce a simple yet effective baseline named self-check, which leverages self-critique to improve task performance for various models. We hope this study serves as an initial exploration into understanding the critique abilities of LLMs, and aims to inform future research, including the development of more proficient critic models and the application of critiques across diverse tasks.

cs.LG↗

arXiv2023

Towards an On-device Agent for Text Rewriting

Large Language Models (LLMs) have demonstrated impressive capabilities for text rewriting. Nonetheless, the large sizes of these models make them impractical for on-device inference, which would otherwise allow for enhanced privacy and economical inference. Creating a smaller yet potent language model for text rewriting presents a formidable challenge because it requires balancing the need for a small size with the need to retain the emergent capabilities of the LLM, that requires costly data collection. To address the above challenge, we introduce a new instruction tuning approach for building a mobile-centric text rewriting model. Our strategies enable the generation of high quality training data without any human labeling. In addition, we propose a heuristic reinforcement learning framework which substantially enhances performance without requiring preference data. To further bridge the performance gap with the larger server-side model, we propose an effective approach that combines the mobile rewrite agent with the server model using a cascade. To tailor the text rewriting tasks to mobile scenarios, we introduce MessageRewriteEval, a benchmark that focuses on text rewriting for messages through natural language instructions. Through empirical experiments, we demonstrate that our on-device model surpasses the current state-of-the-art LLMs in text rewriting while maintaining a significantly reduced model size. Notably, we show that our proposed cascading approach improves model performance.

cs.CL↗

arXiv2023

Class-level Structural Relation Modelling and Smoothing for Visual Representation Learning

Representation learning for images has been advanced by recent progress in more complex neural models such as the Vision Transformers and new learning theories such as the structural causal models. However, these models mainly rely on the classification loss to implicitly regularize the class-level data distributions, and they may face difficulties when handling classes with diverse visual patterns. We argue that the incorporation of the structural information between data samples may improve this situation. To achieve this goal, this paper presents a framework termed \textbf{C}lass-level Structural Relation Modeling and Smoothing for Visual Representation Learning (CSRMS), which includes the Class-level Relation Modelling, Class-aware Graph Sampling, and Relational Graph-Guided Representation Learning modules to model a relational graph of the entire dataset and perform class-aware smoothing and regularization operations to alleviate the issue of intra-class visual diversity and inter-class similarity. Specifically, the Class-level Relation Modelling module uses a clustering algorithm to learn the data distributions in the feature space and identify three types of class-level sample relations for the training set; Class-aware Graph Sampling module extends typical training batch construction process with three strategies to sample dataset-level sub-graphs; and Relational Graph-Guided Representation Learning module employs a graph convolution network with knowledge-guided smoothing operations to ease the projection from different visual patterns to the same class. Experiments demonstrate the effectiveness of structured knowledge modelling for enhanced representation learning and show that CSRMS can be incorporated with any state-of-the-art visual representation learning models for performance gains. The source codes and demos have been released at https://github.com/czt117/CSRMS.

cs.CV↗

arXiv2023

Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data

Federated Learning aims to learn a global model on the server side that generalizes to all clients in a privacy-preserving manner, by leveraging the local models from different clients. Existing solutions focus on either regularizing the objective functions among clients or improving the aggregation mechanism for the improved model generalization capability. However, their performance is typically limited by the dataset biases, such as the heterogeneous data distributions and the missing classes. To address this issue, this paper presents a cross-silo prototypical calibration method (FedCSPC), which takes additional prototype information from the clients to learn a unified feature space on the server side. Specifically, FedCSPC first employs the Data Prototypical Modeling (DPM) module to learn data patterns via clustering to aid calibration. Subsequently, the cross-silo prototypical calibration (CSPC) module develops an augmented contrastive learning method to improve the robustness of the calibration, which can effectively project cross-source features into a consistent space while maintaining clear decision boundaries. Moreover, the CSPC module's ease of implementation and plug-and-play characteristics make it even more remarkable. Experiments were conducted on four datasets in terms of performance comparison, ablation study, in-depth analysis and case study, and the results verified that FedCSPC is capable of learning the consistent features across different data sources of the same class under the guidance of calibrated model, which leads to better performance than the state-of-the-art methods. The source codes have been released at https://github.com/qizhuang-qz/FedCSPC.

cs.LG↗

arXiv2023

Electron and Muon Anomalous Magnetic Moment in the $\mathbb{Z}_3$-NMSSM

Inspired by the recent measurements of the muon and electron anomalous magnetic moments, the rapid progress of the LHC search for supersymmetry, and the significantly improved sensitivities of dark matter direct detection experiments, we studied the supersymmetric contribution to the electron \texorpdfstring{$g-2$}{}, $a_e^{\rm SUSY}$, in the Next-to-Minimal Supersymmetric Standard Model with a discrete $\mathbb{Z}_3$ symmetry. We concluded that $a_e^{\rm SUSY}$ was mainly correlated with $a_μ^{\rm SUSY}$ by the formula $a_e^{\rm SUSY}/m_e^2 \simeq a_μ^{\rm SUSY}/m_μ^2$, and significant violations of this correlation might occur only in rare cases. As a result, $a_e^{\rm SUSY}$ was typically around $5 \times 10^{-14}$ when $a_μ^{\rm SUSY} \simeq 2.5 \times 10^{-9}$. We also concluded that the dark matter direct detection and LHC experiments played crucial roles in determining the maximum reach of $a_e^{\rm SUSY}$. Concretely, $a_e^{\rm SUSY}$ might be around $3 \times 10^{-13}$ in the optimum cases if one used the XENON-1T experiment to limit the supersymmetry parameter space. This prediction, however, was reduced to $1.5 \times 10^{-13}$ after implementing the LZ restrictions and $1.0 \times 10^{-13}$ when further considering the LHC restrictions.

hep-ph↗

arXiv2023

Triple Sequence Learning for Cross-domain Recommendation

Cross-domain recommendation (CDR) aims to leverage the correlation of users' behaviors in both the source and target domains to improve the user preference modeling in the target domain. Conventional CDR methods typically explore the dual-relations between the source and target domains' behaviors. However, this may ignore the informative mixed behaviors that naturally reflect the user's global preference. To address this issue, we present a novel framework, termed triple sequence learning for cross-domain recommendation (Tri-CDR), which jointly models the source, target, and mixed behavior sequences to highlight the global and target preference and precisely model the triple correlation in CDR. Specifically, Tri-CDR independently models the hidden representations for the triple behavior sequences and proposes a triple cross-domain attention (TCA) method to emphasize the informative knowledge related to both user's global and target-domain preference. To comprehensively explore the cross-domain correlations, we design a triple contrastive learning (TCL) strategy that simultaneously considers the coarse-grained similarities and fine-grained distinctions among the triple sequences, ensuring the alignment while preserving information diversity in multi-domain. We conduct extensive experiments and analyses on six cross-domain settings. The significant improvements of Tri-CDR with different sequential encoders verify its effectiveness and universality. The code will be released upon acceptance.

cs.IR↗

arXiv2023

Status of the singlino-dominated dark matter in general Next-to-Minimal Supersymmetric Standard Model

With the rapid progress of dark matter direct detection experiments, the attractiveness of the popular bino-dominated dark matter in economical supersymmetric theories is fading. As an alternative, the singlino-dominated dark matter in general Next-to-Minimal Supersymmetric Standard Model (NMSSM) is paying due attention. This scenario has the following distinct characteristics: free from the tadpole problem and the domain-wall problem of the NMSSM with a $Z_3$-symmetry, predicting more stable vacuum states than the $Z_3$-NMSSM, capable of forming an economical secluded dark matter sector to yield the dark matter experimental results naturally, and readily weaken the restrictions from the LHC search for SUSY. Consequently, it can explain the muon g-2 anomaly in broad parameter space that agrees with various experimental results while simultaneously breaking the electroweak symmetry naturally. In this study, we show in detail how the scenario coincides with the experiments, such as the SUSY search at the LHC, the dark matter search by the LZ experiment, and the improved measurement of the muon g-2. We provide a simple and clear picture of the physics inherent in the general NMSSM.

hep-ph↗

arXiv2022

Interpreting the $W$ mass anomaly in the vector-like quark models

The new measurement of $W$-boson mass by the CDF collaboration revealed a remarkable $7σ$ disagreement with the Standard Model (SM) prediction. If confirmed by other experiments, then the disagreement strongly indicates the existence of new physics beyond the SM. In this work, seven vectorlike quark (VLQ) extensions of the SM are investigated to interpret the anomaly, and it is found that three can explain the anomaly in broad parameter space. The explanations are consistent with the constraints from oblique parameters, the LHC search for VLQs, the measurements of the properties for the top quark, bottom quark, and Higgs boson, and the perturbativity criterion. The typical size of the involved Yukawa coupling is around 1, which is comparable to the top quark Yukawa coupling in the SM. The other extensions, however, either predict a negative correction to the mass in reasonable parameter space or explain the anomaly by unnatural theoretical input parameters.

hep-ph↗

arXiv2022

Multi-modal Video Chapter Generation

Chapter generation becomes practical technique for online videos nowadays. The chapter breakpoints enable users to quickly find the parts they want and get the summative annotations. However, there is no public method and dataset for this task. To facilitate the research along this direction, we introduce a new dataset called Chapter-Gen, which consists of approximately 10k user-generated videos with annotated chapter information. Our data collection procedure is fast, scalable and does not require any additional manual annotation. On top of this dataset, we design an effective baseline specificlly for video chapters generation task. which captures two aspects of a video,including visual dynamics and narration text. It disentangles local and global video features for localization and title generation respectively. To parse the long video efficiently, a skip sliding window mechanism is designed to localize potential chapters. And a cross attention multi-modal fusion module is developed to aggregate local features for title generation. Our experiments demonstrate that the proposed framework achieves superior results over existing methods which illustrate that the method design for similar task cannot be transfered directly even after fine-tuning. Code and dataset are available at https://github.com/czt117/MVCG.

cs.CV↗

arXiv2022

Interpreting the $W$-mass anomaly in the vectorlike quark models

hep-ph↗

arXiv2022

Meta-Causal Feature Learning for Out-of-Distribution Generalization

Causal inference has become a powerful tool to handle the out-of-distribution (OOD) generalization problem, which aims to extract the invariant features. However, conventional methods apply causal learners from multiple data splits, which may incur biased representation learning from imbalanced data distributions and difficulty in invariant feature learning from heterogeneous sources. To address these issues, this paper presents a balanced meta-causal learner (BMCL), which includes a balanced task generation module (BTG) and a meta-causal feature learning module (MCFL). Specifically, the BTG module learns to generate balanced subsets by a self-learned partitioning algorithm with constraints on the proportions of sample classes and contexts. The MCFL module trains a meta-learner adapted to different distributions. Experiments conducted on NICO++ dataset verified that BMCL effectively identifies the class-invariant visual regions for classification and may serve as a general framework to improve the performance of the state-of-the-art methods.

cs.CV↗

arXiv2020

Pairwise versus multiple network alignment

Biological network alignment (NA) aims to identify similar regions between molecular networks of different species. NA can be local or global. Just as the recent trend in the NA field, we also focus on global NA, which can be pairwise (PNA) and multiple (MNA). PNA produces aligned node pairs between two networks. MNA produces aligned node clusters between more than two networks. Recently, the focus has shifted from PNA to MNA, because MNA captures conserved regions between more networks than PNA (and MNA is thus considered to be more insightful), though at higher computational complexity. The issue is that, due to the different outputs of PNA and MNA, a PNA method is only compared to other PNA methods, and an MNA method is only compared to other MNA methods. Comparison of PNA against MNA must be done to evaluate whether MNA's higher complexity is justified by its higher accuracy. We introduce a framework that allows for this. We evaluate eight prominent PNA and MNA methods, on synthetic and real-world biological networks, using topological and functional alignment quality measures. We compare PNA against MNA in both a pairwise (native to PNA) and multiple (native to MNA) manner. PNA is expected to perform better under the pairwise evaluation framework. Indeed this is what we find. MNA is expected to perform better under the multiple evaluation framework. Shockingly, we find this not to always hold; PNA is often better than MNA in this framework, depending on the choice of evaluation test.

q-bio.MN↗

arXiv2020

Anomalous Muon Magnetic Moment in the Inverse Seesaw Extended Next-to-Minimal Supersymmetric Standard Model

The present work investigates the possibility that both dark matter and the anomalous magnetic moment of the muon may be explained within the context of the inverse seesaw extended Next-to-Minimal Supersymmetric Standard Model (ISS-NMSSM). In ISS-NMSSM, the newly introduced Higgs-neutrino Yukawa coupling $Y_ν$ provides additional Higgsino-sneutrino loop contribution to $(g-2)_μ$. If the deviation between the experimental observations and the Standard Model predictions of the anomalous muon magnetic moment is confirmed by the further experimental and theoretical studies, it can be explained naturally within the ISS-NMSSM framework without conflicting with the current stringent limits on the direct detection of dark matter and Large Hadron Collider searches.

hep-ph↗