arXiv subjects

Binghui Guo

Publications and source records attributed to Binghui Guo.

9 recordsLinked to original sources

arXiv2025

EntropyLong: Effective Long-Context Training via Predictive Uncertainty

Training long-context language models to capture long-range dependencies requires specialized data construction. Current approaches, such as generic text concatenation or heuristic-based variants, frequently fail to guarantee genuine long-range dependencies. We propose EntropyLong, a novel data construction method that leverages predictive uncertainty to verify dependency quality. Our approach identifies high-entropy positions in documents, retrieves semantically relevant contexts from large corpora, and verifies their utility by assessing whether they reduce prediction entropy. This model-in-the-loop verification ensures each dependency represents measurable information gain rather than spurious correlation. We construct training samples with long-range dependencies by combining original documents with these verified contextual supplements. Using FineWebEdu and Cosmopedia, we generate a dataset of 128K-length sequences with verified dependencies. Models trained on this data demonstrate significant improvements on RULER benchmarks, particularly in tasks requiring distant information. Following instruction fine-tuning, our models also achieve substantial gains on LongBenchv2, demonstrating enhanced long-context understanding. Extensive ablation studies further validate the necessity and effectiveness of entropybased verification for long-context training.

cs.CL↗

arXiv2025

LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs

High-quality long-context data is essential for training large language models (LLMs) capable of processing extensive documents, yet existing synthesis approaches using relevance-based aggregation face challenges of computational efficiency. We present LiteLong, a resource-efficient method for synthesizing long-context data through structured topic organization and multi-agent debate. Our approach leverages the BISAC book classification system to provide a comprehensive hierarchical topic organization, and then employs a debate mechanism with multiple LLMs to generate diverse, high-quality topics within this structure. For each topic, we use lightweight BM25 retrieval to obtain relevant documents and concatenate them into 128K-token training samples. Experiments on HELMET and Ruler benchmarks demonstrate that LiteLong achieves competitive long-context performance and can seamlessly integrate with other long-dependency enhancement methods. LiteLong makes high-quality long-context data synthesis more accessible by reducing both computational and data engineering costs, facilitating further research in long-context language training.

cs.CL↗

arXiv2025

GPMFS: Global Foundation and Personalized Optimization for Multi-Label Feature Selection

As artificial intelligence methods are increasingly applied to complex task scenarios, high dimensional multi-label learning has emerged as a prominent research focus. At present, the curse of dimensionality remains one of the major bottlenecks in high-dimensional multi-label learning, which can be effectively addressed through multi-label feature selection methods. However, existing multi-label feature selection methods mostly focus on identifying global features shared across all labels, which overlooks personalized characteristics and specific requirements of individual labels. This global-only perspective may limit the ability to capture label-specific discriminative information, thereby affecting overall performance. In this paper, we propose a novel method called GPMFS (Global Foundation and Personalized Optimization for Multi-Label Feature Selection). GPMFS firstly identifies global features by exploiting label correlations, then adaptively supplements each label with a personalized subset of discriminative features using a threshold-controlled strategy. Experiments on multiple real-world datasets demonstrate that GPMFS achieves superior performance while maintaining strong interpretability and robustness. Furthermore, GPMFS provides insights into the label-specific strength across different multi-label datasets, thereby demonstrating the necessity and potential applicability of personalized feature selection approaches.

cs.LG↗

arXiv2025

Learning to Erase Private Knowledge from Multi-Documents for Retrieval-Augmented Large Language Models

Retrieval-Augmented Generation (RAG) is a promising technique for applying LLMs to proprietary domains. However, retrieved documents may contain sensitive knowledge, posing risks of privacy leakage in generative results. Thus, effectively erasing private information from retrieved documents is a key challenge for RAG. Unlike traditional text anonymization, RAG should consider: (1) the inherent multi-document reasoning may face de-anonymization attacks; (2) private knowledge varies by scenarios, so users should be allowed to customize which information to erase; (3) preserving sufficient publicly available knowledge for generation tasks. This paper introduces the privacy erasure task for RAG and proposes Eraser4RAG, a private knowledge eraser which effectively removes user-defined private knowledge from documents while preserving sufficient public knowledge for generation. Specifically, we first construct a global knowledge graph to identify potential knowledge across documents, aiming to defend against de-anonymization attacks. Then we randomly split it into private and public sub-graphs, and fine-tune Flan-T5 to rewrite the retrieved documents excluding private triples. Finally, PPO algorithm optimizes the rewriting model to minimize private triples and maximize public triples retention. Experiments on four QA datasets demonstrate that Eraser4RAG achieves superior erase performance than GPT-4o.

cs.CL↗

arXiv2024

MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models

In a real-world RAG system, the current query often involves spoken ellipses and ambiguous references from dialogue contexts, necessitating query rewriting to better describe user's information needs. However, traditional context-based rewriting has minimal enhancement on downstream generation tasks due to the lengthy process from query rewriting to response generation. Some researchers try to utilize reinforcement learning with generation feedback to assist the rewriter, but these sparse rewards provide little guidance in most cases, leading to unstable training and generation results. We find that user's needs are also reflected in the gold document, retrieved documents and ground truth. Therefore, by feeding back these multi-aspect dense rewards to query rewriting, more stable and satisfactory responses can be achieved. In this paper, we propose a novel query rewriting method MaFeRw, which improves RAG performance by integrating multi-aspect feedback from both the retrieval process and generated results. Specifically, we first use manual data to train a T5 model for the rewriter initialization. Next, we design three metrics as reinforcement learning feedback: the similarity between the rewritten query and the gold document, the ranking metrics, and ROUGE between the generation and the ground truth. Inspired by RLAIF, we train three kinds of reward models for the above metrics to achieve more efficient training. Finally, we combine the scores of these reward models as feedback, and use PPO algorithm to explore the optimal query rewriting strategy. Experimental results on two conversational RAG datasets demonstrate that MaFeRw achieves superior generation metrics and more stable training compared to baselines.

cs.CL↗

arXiv2024

Defending Against Sophisticated Poisoning Attacks with RL-based Aggregation in Federated Learning

Federated learning is highly susceptible to model poisoning attacks, especially those meticulously crafted for servers. Traditional defense methods mainly focus on updating assessments or robust aggregation against manually crafted myopic attacks. When facing advanced attacks, their defense stability is notably insufficient. Therefore, it is imperative to develop adaptive defenses against such advanced poisoning attacks. We find that benign clients exhibit significantly higher data distribution stability than malicious clients in federated learning in both CV and NLP tasks. Therefore, the malicious clients can be recognized by observing the stability of their data distribution. In this paper, we propose AdaAggRL, an RL-based Adaptive Aggregation method, to defend against sophisticated poisoning attacks. Specifically, we first utilize distribution learning to simulate the clients' data distributions. Then, we use the maximum mean discrepancy (MMD) to calculate the pairwise similarity of the current local model data distribution, its historical data distribution, and global model data distribution. Finally, we use policy learning to adaptively determine the aggregation weights based on the above similarities. Experiments on four real-world datasets demonstrate that the proposed defense model significantly outperforms widely adopted defense models for sophisticated attacks.

cs.LG↗

arXiv2015

Organization mechanism and counting algorithm on Vertex-Cover solutions

Counting the solution number of combinational optimization problems is an important topic in the study of computational complexity, especially on the #P-complete complexity class. In this paper, we first investigate some organizations of Vertex-Cover unfrozen subgraphs by the underlying connectivity and connected components of unfrozen vertices. Then, a Vertex-Cover Solution Number Counting Algorithm is proposed and its complexity analysis is provided, the results of which fit very well with the simulations and have better performance than those by 1-RSB in a neighborhood of c = e for random graphs. Base on the algorithm, variation and fluctuation on the solution number statistics are studied to reveal the evolution mechanism of the solution numbers. Besides, marginal probability distributions on the solution space are investigated on both random graph and scale-free graph to illustrate different evolution characteristics of their solution spaces. Thus, doing solution number counting based on graph expression of solution space should be an alternative and meaningful way to study the hardness of NP-complete and #P-complete problems, and appropriate algorithm design can help to achieve better approximations of solving combinational optimization problems and the corresponding counting problems.

cs.CC↗

arXiv2012

Analysis on the evolution process of BFW-like model with explosive percolation of multiple giant components

Recently, the modified BFW model on random graph [Phys. Rev. Lett., 106, 115701 (2011)], which shows a strongly discontinuous percolation transition with multiple giant components, has attracted much attention from physicists, statisticians and materials scientists. In this paper, by establishing the theoretical expression of evolution equations on the modified BFW model, the steady-state and evolution process are analyzed and a close correspondence is built between the values of parameter αand the number of giant components in steady-states, which fits very well with the numerical simulations. In fact, with the value of αdecreasing to 0.25, the error between theoretical and numerical results is smaller than 4% and trends to 0 rapidly. Furthermore, the sizes of giant components for different evolution strategies can also be obtained by solving some constraints derived from the evolution equations. The analysis of the steady-state and evolution process is of great help to explain why the percolation of modified BFW model is explosive and how explosive it is.

math-ph↗

arXiv2012

Determining the Solution Space of Vertex-Cover by Interactions and Backbones

To solve the combinatorial optimization problems especially the minimal Vertex-cover problem with high efficiency, is a significant task in theoretical computer science and many other subjects. Aiming at detecting the solution space of Vertex-cover, a new structure named interaction between nodes is defined and discovered for random graph, which results in the emergence of the frustration and long-range correlation phenomenon. Based on the backbones and interactions with a node adding process, we propose an Interaction and Backbone Evolution Algorithm to achieve the reduced solution graph, which has a direct correspondence to the solution space of Vertex-cover. By this algorithm, the whole solution space can be obtained strictly when there is no leaf-removal core on the graph and the odd cycles of unfrozen nodes bring great obstacles to its efficiency. Besides, this algorithm possesses favorable exactness and has good performance on random instances even with high average degrees. The interaction with the algorithm provides a new viewpoint to solve Vertex-cover, which will have a wide range of applications to different types of graphs, better usage of which can lower the computational complexity for solving Vertex-cover.

math-ph↗