arXiv subjects

Mingqi Li

Publications and source records attributed to Mingqi Li.

9 recordsLinked to original sources

arXiv2025

Hybrid Matrix Factorization Based Graph Contrastive Learning for Recommendation System

In recent years, methods that combine contrastive learning with graph neural networks have emerged to address the challenges of recommendation systems, demonstrating powerful performance and playing a significant role in this domain. Contrastive learning primarily tackles the issue of data sparsity by employing data augmentation strategies, effectively alleviating this problem and showing promising results. Although existing research has achieved favorable outcomes, most current graph contrastive learning methods are based on two types of data augmentation strategies: the first involves perturbing the graph structure, such as by randomly adding or removing edges; and the second applies clustering techniques. We believe that the interactive information obtained through these two strategies does not fully capture the user-item interactions. In this paper, we propose a novel method called HMFGCL (Hybrid Matrix Factorization Based Graph Contrastive Learning), which integrates two distinct matrix factorization techniques-low-rank matrix factorization (MF) and singular value decomposition (SVD)-to complementarily acquire global collaborative information, thereby constructing enhanced views. Experimental results on multiple public datasets demonstrate that our model outperforms existing baselines, particularly on small-scale datasets.

cs.IR↗

arXiv2025

Optimizing Frequent Checkpointing via Low-Cost Differential for Distributed Training Systems

Distributed training of large deep-learning models often leads to failures, so checkpointing is commonly employed for recovery. State-of-the-art studies focus on frequent checkpointing for fast recovery from failures. However, frequent checkpointing generates numerous checkpoints, incurring substantial costs and thus degrading training performance. Recently, differential checkpointing has been proposed to reduce costs, but it is limited to recommendation systems, so its application to general distributed training systems remains unexplored. In this paper, we find that gradients generated during distributed training can be reused to construct differential checkpoints, while the former's size is smaller than the latter's, motivating us to reuse gradients for low-cost differential checkpointing. Based on this main idea, we propose \sysname, a frequent checkpointing framework for compression-enabled training systems that reuses compressed gradients as differential checkpoints, eliminating redundant differential computation and reducing checkpoint transmission cost. Furthermore, we extend gradient reuse to scenarios without gradient compression and propose \sysnameplus, which employs layer-wise-reuse snapshotting and incremental-merging persistence to overlap checkpointing with training execution. Experiments on diverse workloads, including billion-parameter-scale models, demonstrate that \sysname and \sysnameplus significantly reduce checkpointing overhead and enable checkpointing at frequencies as high as once per iteration, reducing training time by up to 89.2\% and 81.2\%, respectively.

cs.DC↗

arXiv2024

Significant Impact of Quantum and Anharmonic Effects on the Structural Stability and Superconductivity of NbH3 at High Pressures

First-principles calculations combined with the stochastic self-consistent harmonic approximation reveal significant effects of the quantum ionic fluctuations and lattice anharmonicity on the dynamical stability of NbH3 under high pressures. Previous theoretical predictions, which ignored ionic fluctuations and relied on the harmonic approximation, suggested that the I43d phase is the most thermodynamically favorable structure between 33 and 400 GPa, with the Fm3m phase considered thermodynamically metastable. However, recent experiments at 187 GPa identified the Fm3m phase, conflicting with the prediction. In contrast, the present study indicates that the Fm3m phase remains dynamically stable down to at least 145 GPa, approximately 145 GPa lower than harmonic estimates, while the I43d phase is dynamically unstable at 187 GPa, consistent with the experimental findings. Furthermore, systematic calculations are performed on the structural, vibrational and superconducting properties of Fm3m NbH3 under pressures ranging from 100 to 300 GPa, revealing dramatic modifications due to the quantum and anharmonic effects. The calculated superconducting critical temperature (Tc) from the McMillan equation for Fm3m NbH3 at 187 GPa is 44 K, with mu set at 0.15, close to the measured value. These findings highlight the crucial role of quantum anharmonic effects in stabilizing the Fm3m phase.

cond-mat.mtrl-sci↗

arXiv2024

Learning from Contrastive Prompts: Automated Optimization and Adaptation

As LLMs evolve, significant effort is spent on manually crafting prompts. While existing prompt optimization methods automate this process, they rely solely on learning from incorrect samples, leading to a sub-optimal performance. Additionally, an unexplored challenge in the literature is prompts effective for prior models may not perform well on newer versions or different languages. We propose the Learning from Contrastive Prompts (LCP) framework to address these gaps, enhancing both prompt optimization and adaptation. LCP employs contrastive learning to generate effective prompts by analyzing patterns in good and bad prompt examples. Our evaluation on the Big-Bench Hard dataset shows that LCP has a win rate of over 76% over existing methods in prompt optimization and demonstrates strong adaptability across different model versions, families, and languages. LCP offers a systematic approach to prompt engineering, reducing manual effort in deploying LLMs across varied contexts.

cs.CL↗

arXiv2024

H3Se in the Im-3m Phase: A High-Pressure Superconductor with Tc Reaching 200 K at 64 GPa Mediated by Anharmonic Phonons

Hydrogen-based compounds have attracted significant attention in recent years due to the discovery of conventional superconductivity with high critical temperature under high pressure, rekindling hopes for searching room temperature superconductor. In this work, we investigated systematically the vibrational and superconducting properties of H3Se in Im-3m phase under pressures ranging from 50 to 200 GPa. Our approach combines the stochastic self-consistent harmonic approximation with first-principles calculations to address effects from the quantum and anharmonic vibrations of ions. It turns out that these effects significantly modify the crystal structure, increasing the inner pressure by about 8 GPa compared to situations where they are ignored. The phonon spectra suggest that with these effects included, the crystal can be stabilized at pressures as low as about 61 GPa, much lower than the previously predicted value of over 100 GPa. Our calculations also highlight the critical role of quantum and anharmonic effects on the electron-phonon coupling properties. Neglecting these factors could result in a substantial overestimation of the superconducting critical temperature Tc, by approximately 25 K at 125 GPa, for example. With anharmonic phonons, the Tc derived from the Migdal-Eliashberg equations, reaches 200 K ($\mu^\star$= 0.1, $\lambda$=4.1) as the pressure decreases to 64 GPa, making the crystal a rare high-Tc superconductor at moderate pressures.

cond-mat.supr-con↗

arXiv2024

Effectively Prompting Small-sized Language Models for Cross-lingual Tasks via Winning Tickets

Current soft prompt methods yield limited performance when applied to small-sized models (fewer than a billion parameters). Deep prompt-tuning, which entails prepending parameters in each layer for enhanced efficacy, presents a solution for prompting small-sized models, albeit requiring carefully designed implementation. In this paper, we introduce the Lottery Ticket Prompt-learning (LTP) framework that integrates winning tickets with soft prompts. The LTP offers a simpler implementation and requires only a one-time execution. We demonstrate LTP on cross-lingual tasks, where prior works rely on external tools like human-designed multilingual templates and bilingual dictionaries, which may not be feasible in a low-resource regime. Specifically, we select a subset of parameters that have been changed the most during the fine-tuning with the Masked Language Modeling objective. Then, we prepend soft prompts to the original pre-trained language model and only update the selected parameters together with prompt-related parameters when adapting to the downstream tasks. We verify the effectiveness of our LTP framework on cross-lingual tasks, specifically targeting low-resource languages. Our approach outperforms the baselines by only updating 20\% of the original parameters.

cs.CL↗

arXiv2022

Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model

Pre-trained multilingual language models play an important role in cross-lingual natural language understanding tasks. However, existing methods did not focus on learning the semantic structure of representation, and thus could not optimize their performance. In this paper, we propose Multi-level Multilingual Knowledge Distillation (MMKD), a novel method for improving multilingual language models. Specifically, we employ a teacher-student framework to adopt rich semantic representation knowledge in English BERT. We propose token-, word-, sentence-, and structure-level alignment objectives to encourage multiple levels of consistency between source-target pairs and correlation similarity between teacher and student models. We conduct experiments on cross-lingual evaluation benchmarks including XNLI, PAWS-X, and XQuAD. Experimental results show that MMKD outperforms other baseline models of similar size on XNLI and XQuAD and obtains comparable performance on PAWS-X. Especially, MMKD obtains significant performance gains on low-resource languages.

cs.CL↗

arXiv2021

Segmentation with Multiple Acceptable Annotations: A Case Study of Myocardial Segmentation in Contrast Echocardiography

Most existing deep learning-based frameworks for image segmentation assume that a unique ground truth is known and can be used for performance evaluation. This is true for many applications, but not all. Myocardial segmentation of Myocardial Contrast Echocardiography (MCE), a critical task in automatic myocardial perfusion analysis, is an example. Due to the low resolution and serious artifacts in MCE data, annotations from different cardiologists can vary significantly, and it is hard to tell which one is the best. In this case, how can we find a good way to evaluate segmentation performance and how do we train the neural network? In this paper, we address the first problem by proposing a new extended Dice to effectively evaluate the segmentation performance when multiple accepted ground truth is available. Then based on our proposed metric, we solve the second problem by further incorporating the new metric into a loss function that enables neural networks to flexibly learn general features of myocardium. Experiment results on our clinical MCE data set demonstrate that the neural network trained with the proposed loss function outperforms those existing ones that try to obtain a unique ground truth from multiple annotations, both quantitatively and qualitatively. Finally, our grading study shows that using extended Dice as an evaluation metric can better identify segmentation results that need manual correction compared with using Dice.

cs.CV↗

arXiv2019

Workflow Scheduling in the Cloud with Weighted Upward-rank Priority Scheme Using Random Walk and Uniform Spare Budget Splitting

We study a difficult problem of how to schedule complex workflows with precedence constraints under a limited budget in the cloud environment. We first formulate the scheduling problem as an integer programming problem, which can be optimized and used as the baseline of performance. We then consider the traditional approach of scheduling jobs in a prioritized order based on the upward-rank of each job. For those jobs with no precedence constraints among themselves, the plain upward-rank priority scheme assigns priorities in an arbitrary way. We propose a job prioritization scheme that uses Markovian chain stationary probabilities as a measure of importance of jobs. The scheme keeps the precedence order for the jobs that have precedence constraints between each other, and assigns priorities according to the jobs' importance for the jobs without precedence constraints. We finally design a uniform spare budget splitting strategy, which splits the spare budget uniformly across all the jobs. We test our algorithms on a variety of workflows, including FFT, Gaussian elimination, typical scientific workflows, randomly generated workflows and workflows from an in-production cluster of an online streaming service company. We compare our algorithms with the-state-of-art algorithms. The empirical results show that the uniform spare budget splitting scheme outperforms the splitting scheme in proportion to extra demand in average for most cases, and the Markovian based prioritization further improves the workflow makespan.

cs.DC↗