arXiv subjects

Rui Li

Publications and source records attributed to Rui Li.

At least 217 records · Page 12Linked to original sources

arXiv2025

WiNGPT-3.0 Technical Report

Current Large Language Models (LLMs) exhibit significant limitations, notably in structured, interpretable, and verifiable medical reasoning, alongside practical deployment challenges related to computational resources and data privacy. This report focused on the development of WiNGPT-3.0, the 32-billion parameter LLMs, engineered with the objective of enhancing its capacity for medical reasoning and exploring its potential for effective integration within healthcare IT infrastructures. The broader aim is to advance towards clinically applicable models. The approach involved a multi-stage training pipeline tailored for general, medical, and clinical reasoning. This pipeline incorporated supervised fine-tuning (SFT) and reinforcement learning (RL), leveraging curated Long Chain-of-Thought (CoT) datasets, auxiliary reward models, and an evidence-based diagnostic chain simulation. WiNGPT-3.0 demonstrated strong performance: specific model variants achieved scores of 66.6 on MedCalc and 87.1 on MedQA-USMLE. Furthermore, targeted training improved performance on a clinical reasoning task from a baseline score of 58.1 to 62.5. These findings suggest that reinforcement learning, even when applied with a limited dataset of only a few thousand examples, can enhance medical reasoning accuracy. Crucially, this demonstration of RL's efficacy with limited data and computation paves the way for more trustworthy and practically deployable LLMs within clinical workflows and health information infrastructures.

cs.CL↗

arXiv2025

Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models

Sentence embedding is essential for many NLP tasks, with contrastive learning methods achieving strong performance using annotated datasets like NLI. Yet, the reliance on manual labels limits scalability. Recent studies leverage large language models (LLMs) to generate sentence pairs, reducing annotation dependency. However, they overlook ranking information crucial for fine-grained semantic distinctions. To tackle this challenge, we propose a method for controlling the generation direction of LLMs in the latent space. Unlike unconstrained generation, the controlled approach ensures meaningful semantic divergence. Then, we refine exist sentence embedding model by integrating ranking information and semantic information. Experiments on multiple benchmarks demonstrate that our method achieves new SOTA performance with a modest cost in ranking sentence synthesis.

cs.CL↗

arXiv2025

Nested Hash Layer: A Plug-and-play Module for Multiple-length Hash Code Learning

Deep supervised hashing is essential for efficient storage and search in large-scale image retrieval. Traditional deep supervised hashing models generate single-length hash codes, but this creates a trade-off between efficiency and effectiveness for different code lengths. To find the optimal length for a task, multiple models must be trained, increasing time and computation. Furthermore, relationships between hash codes of different lengths are often ignored. To address these issues, we propose the Nested Hash Layer (NHL), a plug-and-play module for deep supervised hashing models. NHL generates hash codes of multiple lengths simultaneously in a nested structure. To resolve optimization conflicts from multiple learning objectives, we introduce a dominance-aware dynamic weighting strategy to adjust gradients. Additionally, we propose a long-short cascade self-distillation method, where long hash codes guide the learning of shorter ones, improving overall code quality. Experiments indicate that the NHL achieves an overall training speed improvement of approximately 5 to 8 times across various deep supervised hashing models and enhances the average performance of these models by about 3.4%.

cs.CV↗

arXiv2025

MGS3: A Multi-Granularity Self-Supervised Code Search Framework

In the pursuit of enhancing software reusability and developer productivity, code search has emerged as a key area, aimed at retrieving code snippets relevant to functionalities based on natural language queries. Despite significant progress in self-supervised code pre-training utilizing the vast amount of code data in repositories, existing methods have primarily focused on leveraging contrastive learning to align natural language with function-level code snippets. These studies have overlooked the abundance of fine-grained (such as block-level and statement-level) code snippets prevalent within the function-level code snippets, which results in suboptimal performance across all levels of granularity. To address this problem, we first construct a multi-granularity code search dataset called MGCodeSearchNet, which contains 536K+ pairs of natural language and code snippets. Subsequently, we introduce a novel Multi-Granularity Self-Supervised contrastive learning code Search framework (MGS$^{3}$}). First, MGS$^{3}$ features a Hierarchical Multi-Granularity Representation module (HMGR), which leverages syntactic structural relationships for hierarchical representation and aggregates fine-grained information into coarser-grained representations. Then, during the contrastive learning phase, we endeavor to construct positive samples of the same granularity for fine-grained code, and introduce in-function negative samples for fine-grained code. Finally, we conduct extensive experiments on code search benchmarks across various granularities, demonstrating that the framework exhibits outstanding performance in code search tasks of multiple granularities. These experiments also showcase its model-agnostic nature and compatibility with existing pre-trained code representation models.

cs.SE↗

arXiv2025

Identifying Compton-thick AGNs with Machine learning algorithm in Chandra Deep Field-South

Compton-thick active galactic nuclei (CT-AGNs), which are defined by column density $\mathrm{N_H} \geqslant 1.5 \times 10^{24} \ \mathrm{cm}^{-2}$, emit feeble X-ray radiation, even undetectable by X-ray instruments. Despite this, the X-ray emissions from CT-AGNs are believed to be a substantial contributor to the cosmic X-ray background (CXB). According to synthesis models of AGNs, CT-AGNs are expected to make up a significant fraction of the AGN population, likely around 30% or more. However, only $\sim$11% of AGNs have been identified as CT-AGNs in the Chandra Deep Field-South (CDFS). To identify hitherto unknown CT-AGNs in the field, we used a Random Forest algorithm for identifying them. First, we build a secure classified subset of 210 AGNs to train and evaluate our algorithm. Our algorithm achieved an accuracy rate of 90% on the test set after training. Then, we applied our algorithm to an additional subset of 254 AGNs, successfully identifying 67 CT-AGNs within this group. This result significantly increased the fraction of CT-AGNs in the CDFS, which is closer to the theoretical predictions of the CXB. Finally, we compared the properties of host galaxies between CT-AGNs and non-CT-AGNs and found that the host galaxies of CT-AGNs exhibit higher levels of star formation activity.

astro-ph.GA↗

arXiv2025

KnowTrace: Bootstrapping Iterative Retrieval-Augmented Generation with Structured Knowledge Tracing

Recent advances in retrieval-augmented generation (RAG) furnish large language models (LLMs) with iterative retrievals of relevant information to handle complex multi-hop questions. These methods typically alternate between LLM reasoning and retrieval to accumulate external information into the LLM's context. However, the ever-growing context inherently imposes an increasing burden on the LLM to perceive connections among critical information pieces, with futile reasoning steps further exacerbating this overload issue. In this paper, we present KnowTrace, an elegant RAG framework to (1) mitigate the context overload and (2) bootstrap higher-quality multi-step reasoning. Instead of simply piling the retrieved contents, KnowTrace autonomously traces out desired knowledge triplets to organize a specific knowledge graph relevant to the input question. Such a structured workflow not only empowers the LLM with an intelligible context for inference, but also naturally inspires a reflective mechanism of knowledge backtracing to identify contributive LLM generations as process supervision data for self-bootstrapping. Extensive experiments show that KnowTrace consistently surpasses existing methods across three multi-hop question answering benchmarks, and the bootstrapped version further amplifies the gains.

cs.CL↗

arXiv2025

Hole subband dispersions and strong `spin'-orbit coupling in a cylindrical Ge nanowire

Quasi-one-dimensional hole gas is achievable in a semiconductor Ge nanowire. The lowest two subband dispersions of the hole gas are just two shifted parabolic curves with an anticrossing at $k_{z}=0$. This peculiar low-energy subband structure manifests the existence of a strong `spin' (pseudo spin)-orbit coupling. Based on the Luttinger-Kohn Hamiltonian in the axial approximation, we show two sets of combined dispersions that not only isolated from each other but also with strong `spin'-orbit coupling are obtainable in the presence of strong magnetic field. Realistic calculations are performed for three representative nanowire growth directions [001], [111], and [110]. These results are further confirmed via constructing the low-energy effective Hamiltonian of the hole gas. We also calculate the external electric field induced spin splitting for comparison with the magnetic field induced spin splitting.

cond-mat.mes-hall↗

arXiv2025

Morpho-Photometric Classification of KiDS DR5 Sources Based on Neural Networks: A Comprehensive Star-Quasar-Galaxy Catalog

We present a novel multimodal neural network (MNN) for classifying astronomical sources in multiband ground-based observations, from optical to near infrared, to separate sources in stars, galaxies and quasars. Our approach combines a convolutional neural network branch for learning morphological features from $r$-band images with an artificial neural network branch for extracting spectral energy distribution (SED) information. Specifically, we have used 9-band optical ($ugri$) and NIR ($ZYHJK_s$) data from the Kilo-Degree Survey (KiDS) Data Release 5. The two branches of the network are concatenated and feed into fully-connected layers for final classification. We train the network on a spectroscopically confirmed sample from the Sloan Digital Sky Survey cross-matched with KiDS. The trained model achieves 98.76\% overall accuracy on an independent testing dataset, with F1 scores exceeding 95\% for each class. Raising the output probability threshold, we obtain higher purity at the cost of a lower completeness. We have also validated the network using external catalogs cross-matched with KiDS, correctly classifying 99.74\% of a pure star sample selected from Gaia parallaxes and proper motions, and 99.74\% of an external galaxy sample from the Galaxy and Mass Assembly survey, adjusted for low-redshift contamination. We apply the trained network to 27,335,836 KiDS DR5 sources with $r \leqslant 23$ mag to generate a new classification catalog. This MNN successfully leverages both morphological and SED information to enable efficient and robust classification of stars, quasars, and galaxies in large photometric surveys.

astro-ph.GA↗

arXiv2025

SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarios

Currently, long-chain reasoning remains a key challenge for large language models (LLMs) because natural texts lack sufficient explicit reasoning data. However, existing benchmarks suffer from limitations such as narrow coverage, short reasoning paths, or high construction costs. We introduce SCoRE (Scenario-based Commonsense Reasoning Evaluation), a benchmark that synthesizes multi-hop questions from scenario schemas of entities, relations, and logical rules to assess long-chain commonsense reasoning. SCoRE contains 100k bilingual (Chinese-English) multiple-choice questions whose reasoning chains span 2-11 hops and are grouped into various difficulty levels. Each question is accompanied by fine-grained knowledge labels, explicit reasoning chains, and difficulty levels for diagnostic evaluation. Evaluation results on cutting-edge LLMs such as o3-mini and Deepseek R1 shows that even the best model attains only 69.78% accuracy on SCoRE (even only 47.91% on the hard set), with errors often stemming from rare knowledge, logical inconsistency, and over-interpretation of simple questions. SCoRE offers a scalable, extensible framework for evaluating and diagnosing the long-chain commonsense reasoning abilities of LLMs and guiding future advances in model design and training.

cs.CL↗

arXiv2025

CursorCore: Assist Programming through Aligning Anything

Large language models have been successfully applied to programming assistance tasks, such as code completion, code insertion, and instructional code editing. However, these applications remain insufficiently automated and struggle to effectively integrate various types of information during the programming process, including coding history, current code, and user instructions. In this work, we propose a new conversational framework that comprehensively integrates these information sources, collect data to train our models and evaluate their performance. Firstly, to thoroughly evaluate how well models align with different types of information and the quality of their outputs, we introduce a new benchmark, APEval (Assist Programming Eval), to comprehensively assess the performance of models in programming assistance tasks. Then, for data collection, we develop a data generation pipeline, Programming-Instruct, which synthesizes training data from diverse sources, such as GitHub and online judge platforms. This pipeline can automatically generate various types of messages throughout the programming process. Finally, using this pipeline, we generate 219K samples, fine-tune multiple models, and develop the CursorCore series. We show that CursorCore outperforms other models of comparable size. This framework unifies applications such as inline chat and automated editing, contributes to the advancement of coding assistants. Code, models and data are freely available at https://github.com/TechxGenus/CursorCore.

cs.CL↗

arXiv2025

Machine-Learning Insights into the Entanglement-trainability Correlation of Parameterized Quantum Circuits

Variational quantum algorithms (VQAs) have emerged as the leading strategy to obtain quantum advantage on the current noisy intermediate-scale devices. However, their entanglement-trainability correlation, as the major reason for the barren plateau (BP) phenomenon, poses a challenge to their applications. In this Letter, we suggest a gate-to-tensor (GTT) encoding method for parameterized quantum circuits (PQCs), with which two long short-term memory networks (L-G networks) are trained to predict both entanglement and trainability. The remarkable capabilities of the L-G networks afford a statistical way to delve into the entanglement-trainability correlation of PQCs within a dataset encompassing millions of instances. This machine-learning-driven method first confirms that the more entanglement, the more possible the BP problem. Then, we observe that there still exist PQCs with both high entanglement and high trainability. Furthermore, the trained L-G networks result in an impressive increase in time efficiency by about one million times when constructing a PQC with specific entanglement and trainability, demonstrating their practical applications in VQAs.

quant-ph↗

arXiv2025

Less is More: Efficient Weight Farcasting with 1-Layer Neural Network

Addressing the computational challenges inherent in training large-scale deep neural networks remains a critical endeavor in contemporary machine learning research. While previous efforts have focused on enhancing training efficiency through techniques such as gradient descent with momentum, learning rate scheduling, and weight regularization, the demand for further innovation continues to burgeon as model sizes keep expanding. In this study, we introduce a novel framework which diverges from conventional approaches by leveraging long-term time series forecasting techniques. Our method capitalizes solely on initial and final weight values, offering a streamlined alternative for complex model architectures. We also introduce a novel regularizer that is tailored to enhance the forecasting performance of our approach. Empirical evaluations conducted on synthetic weight sequences and real-world deep learning architectures, including the prominent large language model DistilBERT, demonstrate the superiority of our method in terms of forecasting accuracy and computational efficiency. Notably, our framework showcases improved performance while requiring minimal additional computational overhead, thus presenting a promising avenue for accelerating the training process across diverse tasks and architectures.

cs.LG↗

arXiv2025

MemEngine: A Unified and Modular Library for Developing Advanced Memory of LLM-based Agents

Recently, large language model based (LLM-based) agents have been widely applied across various fields. As a critical part, their memory capabilities have captured significant interest from both industrial and academic communities. Despite the proposal of many advanced memory models in recent research, however, there remains a lack of unified implementations under a general framework. To address this issue, we develop a unified and modular library for developing advanced memory models of LLM-based agents, called MemEngine. Based on our framework, we implement abundant memory models from recent research works. Additionally, our library facilitates convenient and extensible memory development, and offers user-friendly and pluggable memory usage. For benefiting our community, we have made our project publicly available at https://github.com/nuster1128/MemEngine.

cs.AI↗

arXiv2025

Sparse2DGS: Geometry-Prioritized Gaussian Splatting for Surface Reconstruction from Sparse Views

We present a Gaussian Splatting method for surface reconstruction using sparse input views. Previous methods relying on dense views struggle with extremely sparse Structure-from-Motion points for initialization. While learning-based Multi-view Stereo (MVS) provides dense 3D points, directly combining it with Gaussian Splatting leads to suboptimal results due to the ill-posed nature of sparse-view geometric optimization. We propose Sparse2DGS, an MVS-initialized Gaussian Splatting pipeline for complete and accurate reconstruction. Our key insight is to incorporate the geometric-prioritized enhancement schemes, allowing for direct and robust geometric learning under ill-posed conditions. Sparse2DGS outperforms existing methods by notable margins while being ${2}\times$ faster than the NeRF-based fine-tuning approach.

cs.CV↗

arXiv2025

The Other Side of the Coin: Exploring Fairness in Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by retrieving relevant document from external knowledge sources. By referencing this external knowledge, RAG effectively reduces the generation of factually incorrect content and addresses hallucination issues within LLMs. Recently, there has been growing attention to improving the performance and efficiency of RAG systems from various perspectives. While these advancements have yielded significant results, the application of RAG in domains with considerable societal implications raises a critical question about fairness: What impact does the introduction of the RAG paradigm have on the fairness of LLMs? To address this question, we conduct extensive experiments by varying the LLMs, retrievers, and retrieval sources. Our experimental analysis reveals that the scale of the LLMs plays a significant role in influencing fairness outcomes within the RAG framework. When the model scale is smaller than 8B, the integration of retrieval mechanisms often exacerbates unfairness in small-scale LLMs (e.g., LLaMA3.2-1B, Mistral-7B, and LLaMA3-8B). To mitigate the fairness issues introduced by RAG for small-scale LLMs, we propose two approaches, FairFT and FairFilter. Specifically, in FairFT, we align the retriever with the LLM in terms of fairness, enabling it to retrieve documents that facilitate fairer model outputs. In FairFilter, we propose a fairness filtering mechanism to filter out biased content after retrieval. Finally, we validate our proposed approaches on real-world datasets, demonstrating their effectiveness in improving fairness while maintaining performance.

cs.CL↗

arXiv2025

CFSSeg: Closed-Form Solution for Class-Incremental Semantic Segmentation of 2D Images and 3D Point Clouds

2D images and 3D point clouds are foundational data types for multimedia applications, including real-time video analysis, augmented reality (AR), and 3D scene understanding. Class-incremental semantic segmentation (CSS) requires incrementally learning new semantic categories while retaining prior knowledge. Existing methods typically rely on computationally expensive training based on stochastic gradient descent, employing complex regularization or exemplar replay. However, stochastic gradient descent-based approaches inevitably update the model's weights for past knowledge, leading to catastrophic forgetting, a problem exacerbated by pixel/point-level granularity. To address these challenges, we propose CFSSeg, a novel exemplar-free approach that leverages a closed-form solution, offering a practical and theoretically grounded solution for continual semantic segmentation tasks. This eliminates the need for iterative gradient-based optimization and storage of past data, requiring only a single pass through new samples per step. It not only enhances computational efficiency but also provides a practical solution for dynamic, privacy-sensitive multimedia environments. Extensive experiments on 2D and 3D benchmark datasets such as Pascal VOC2012, S3DIS, and ScanNet demonstrate CFSSeg's superior performance.

cs.CV↗

arXiv2025

Subband structure of a cylindrical HgTe nanowire: transition from normal type to inverted type

Based on the $6\times6$ Kane model in the spherical approximation, both the electron and the hole subband dispersions in a cylindrical HgTe nanowire are calculated using analytical method. The transcendental equations determining the subband energies in the nanowire are analytically derived by utilizing both the total angular momentum conservation and the hard wall boundary condition. We show there exists topological band transition in the HgTe nanowire, very similar to that demonstrated in a HgTe quantum well. Via tuning the radius of the nanowire, we find there is a gap-closing-and-reopening transition in the subband structure. When the radius is larger than the critical radius $R_{\rm c}\approx3.2$ nm, the HgTe nanowire is in the inverted regime.

cond-mat.mes-hall↗

arXiv2025

Constraints on Velocity and Spin Dependent Exotic Interaction at the Millimeter Scale with a Diamagnetic-levitated Force Sensor

Light bosons, beyond the standard model and as prominent candidates for dark matter, can mediate velocity and spin dependent exotic interaction between electron spins and nucleons. At short ranges, it remains an open challenge to test this exotic interaction with high precision. Here, we present a method based on diamagnetic-levitated force sensor to detect the exotic interaction at the millimeter scale. Improved constraints for the coupling $g_A^e g_V^N$ are established, within the force range spanning from 0.15 mm to 1.5 mm. And the constraint $|g_A^e g_V^N| \leq 4.39 \times10^{-26}$ at $λ$ = 0.5 mm at the $95 \%$ confidence level, significantly surpasses previous results by more than three orders of magnitude. The diamagnetic levitation force measurement system developed here can also be repurposed to probe other exotic spin-dependent interactions, such as the exotic spin-spin interaction. This system provides a platform for the exploration of dark matter.

hep-ph↗