arXiv subjects

Zhongyang Li

Publications and source records attributed to Zhongyang Li.

At least 37 records · Page 2Linked to original sources

arXiv2025

A Survey of Personalized Large Language Models: Progress and Future Directions

Large Language Models (LLMs) excel in handling general knowledge tasks, yet they struggle with user-specific personalization, such as understanding individual emotions, writing styles, and preferences. Personalized Large Language Models (PLLMs) tackle these challenges by leveraging individual user data, such as user profiles, historical dialogues, content, and interactions, to deliver responses that are contextually relevant and tailored to each user's specific needs. This is a highly valuable research topic, as PLLMs can significantly enhance user satisfaction and have broad applications in conversational agents, recommendation systems, emotion recognition, medical assistants, and more. This survey reviews recent advancements in PLLMs from three technical perspectives: prompting for personalized context (input level), finetuning for personalized adapters (model level), and alignment for personalized preferences (objective level). To provide deeper insights, we also discuss current limitations and outline several promising directions for future research. Updated information about this survey can be found at the https://github.com/JiahongLiu21/Awesome-Personalized-Large-Language-Models.

cs.AI↗

arXiv2025

MemGuide: Intent-Driven Memory Selection for Goal-Oriented Multi-Session LLM Agents

Modern task-oriented dialogue (TOD) systems increasingly rely on large language model (LLM) agents, leveraging Retrieval-Augmented Generation (RAG) and long-context capabilities for long-term memory utilization. However, these methods are primarily based on semantic similarity, overlooking task intent and reducing task coherence in multi-session dialogues. To address this challenge, we introduce MemGuide, a two-stage framework for intent-driven memory selection. (1) Intent-Aligned Retrieval matches the current dialogue context with stored intent descriptions in the memory bank, retrieving QA-formatted memory units that share the same goal. (2) Missing-Slot Guided Filtering employs a chain-of-thought slot reasoner to enumerate unfilled slots, then uses a fine-tuned LLaMA-8B filter to re-rank the retrieved units by marginal slot-completion gain. The resulting memory units inform a proactive strategy that minimizes conversational turns by directly addressing information gaps. Based on this framework, we introduce the MS-TOD, the first multi-session TOD benchmark comprising 132 diverse personas, 956 task goals, and annotated intent-aligned memory targets, supporting efficient multi-session task completion. Evaluations on MS-TOD show that MemGuide raises the task success rate by 11% (88% -> 99%) and reduces dialogue length by 2.84 turns in multi-session settings, while maintaining parity with single-session benchmarks.

cs.CL↗

arXiv2025

A Survey on Transformer Context Extension: Approaches and Evaluation

Large language models (LLMs) based on Transformer have been widely applied in the filed of natural language processing (NLP), demonstrating strong performance, particularly in handling short text tasks. However, when it comes to long context scenarios, the performance of LLMs degrades due to some challenges. To alleviate this phenomenon, there is a number of work proposed recently. In this survey, we first list the challenges of applying pre-trained LLMs to process long contexts. Then systematically review the approaches related to long context and propose our taxonomy categorizing them into four main types: positional encoding, context compression, retrieval augmented, and attention pattern. In addition to the approaches, we focus on the evaluation of long context, organizing relevant data, tasks, and metrics based on existing long context benchmarks. Finally, we summarize unresolved issues in the long context domain and put forward our views on future developments.

cs.CL↗

arXiv2025

GeAR: Graph-enhanced Agent for Retrieval-augmented Generation

Retrieval-augmented Generation (RAG) relies on effective retrieval capabilities, yet traditional sparse and dense retrievers inherently struggle with multi-hop retrieval scenarios. In this paper, we introduce GeAR, a system that advances RAG performance through two key innovations: (i) an efficient graph expansion mechanism that augments any conventional base retriever, such as BM25, and (ii) an agent framework that incorporates the resulting graph-based retrieval into a multi-step retrieval framework. Our evaluation demonstrates GeAR's superior retrieval capabilities across three multi-hop question answering datasets. Notably, our system achieves state-of-the-art results with improvements exceeding 10% on the challenging MuSiQue dataset, while consuming fewer tokens and requiring fewer iterations than existing multi-step retrieval systems. The project page is available at https://gear-rag.github.io.

cs.CL↗

arXiv2025

MemEngine: A Unified and Modular Library for Developing Advanced Memory of LLM-based Agents

Recently, large language model based (LLM-based) agents have been widely applied across various fields. As a critical part, their memory capabilities have captured significant interest from both industrial and academic communities. Despite the proposal of many advanced memory models in recent research, however, there remains a lack of unified implementations under a general framework. To address this issue, we develop a unified and modular library for developing advanced memory models of LLM-based agents, called MemEngine. Based on our framework, we implement abundant memory models from recent research works. Additionally, our library facilitates convenient and extensible memory development, and offers user-friendly and pluggable memory usage. For benefiting our community, we have made our project publicly available at https://github.com/nuster1128/MemEngine.

cs.AI↗

arXiv2025

C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization for Test-Time Expert Re-Mixing

Mixture-of-Experts (MoE) Large Language Models (LLMs) suffer from severely sub-optimal expert pathways-our study reveals that naive expert selection learned from pretraining leaves a surprising 10-20% accuracy gap for improvement. Motivated by this observation, we develop a novel class of test-time optimization methods to re-weight or "re-mixing" the experts in different layers jointly for each test sample. Since the test sample's ground truth is unknown, we propose to optimize a surrogate objective defined by the sample's "successful neighbors" from a reference set of samples. We introduce three surrogates and algorithms based on mode-finding, kernel regression, and the average loss of similar reference samples/tasks. To reduce the cost of optimizing whole pathways, we apply our algorithms merely to the core experts' mixing weights in critical layers, which enjoy similar performance but save significant computation. This leads to "Critical-Layer, Core-Expert, Collaborative Pathway Optimization (C3PO)". We apply C3PO to two recent MoE LLMs and examine it on six widely-used benchmarks. It consistently improves the base model by 7-15% in accuracy and outperforms widely used test-time learning baselines, e.g., in-context learning and prompt/prefix tuning, by a large margin. Moreover, C3PO enables MoE LLMs with 1-3B active parameters to outperform LLMs of 7-9B parameters, hence improving MoE's advantages on efficiency. Our thorough ablation study further sheds novel insights on achieving test-time improvement on MoE.

cs.LG↗

arXiv2025

Critical site percolation and cutsets

In 2003, Kahn conjectured a characterization of the critical percolation probability $p_c$ in terms of vertex cut sets (\cite{JK03}). Later, Lyons and Peres (2016) conjectured a similar characterization of $p_c$ , but in terms of edge cut sets (\cite{LP16}). Both conjectures were subsequently proven by Tang (\cite{pt23}) for bond percolation and site percolation on bounded-degree graphs. Tang further conjectured that Kahn's vertex-cut characterization for $p_c^{site}$ and the Lyons-Peres edge-cut characterization for $p_c^{site}$ would hold for site percolation on any infinite, connected, locally finite graph. In this paper, we establish Kahn's vertex-cut characterization for $p_c^{site}$ by adapting arguments from \cite{jmh57a, DCT15}. Additionally, we disprove the Lyons-Peres edge-cut characterization for $p_c^{site}$ by constructing a counterexample.

math.PR↗

arXiv2025

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts

In large multimodal models (LMMs), the perception of non-language modalities (e.g., visual representations) is usually not on par with the large language models (LLMs)' powerful reasoning capabilities, deterring LMMs' performance on challenging downstream tasks. This weakness has been recently mitigated by replacing the vision encoder with a mixture-of-experts (MoE), which provides rich, multi-granularity, and diverse representations required by diverse downstream tasks. The performance of multimodal MoE largely depends on its router, which reweights and mixes the representations of different experts for each input. However, we find that the end-to-end trained router does not always produce the optimal routing weights for every test sample. To bridge the gap, we propose a novel and efficient method "Re-Routing in Test-Time (R2-T2)" that locally optimizes the vector of routing weights in test-time by moving it toward those vectors of the correctly predicted samples in a neighborhood of the test sample. We propose three R2-T2 strategies with different optimization objectives and neighbor-search spaces. R2-T2 consistently and greatly improves state-of-the-art LMMs' performance on challenging benchmarks of diverse tasks, without training any base-model parameters.

cs.LG↗

arXiv2025

VAGeo: View-specific Attention for Cross-View Object Geo-Localization

Cross-view object geo-localization (CVOGL) aims to locate an object of interest in a captured ground- or drone-view image within the satellite image. However, existing works treat ground-view and drone-view query images equivalently, overlooking their inherent viewpoint discrepancies and the spatial correlation between the query image and the satellite-view reference image. To this end, this paper proposes a novel View-specific Attention Geo-localization method (VAGeo) for accurate CVOGL. Specifically, VAGeo contains two key modules: view-specific positional encoding (VSPE) module and channel-spatial hybrid attention (CSHA) module. In object-level, according to the characteristics of different viewpoints of ground and drone query images, viewpoint-specific positional codings are designed to more accurately identify the click-point object of the query image in the VSPE module. In feature-level, a hybrid attention in the CSHA module is introduced by combining channel attention and spatial attention mechanisms simultaneously for learning discriminative features. Extensive experimental results demonstrate that the proposed VAGeo gains a significant performance improvement, i.e., improving acc@0.25/acc@0.5 on the CVOGL dataset from 45.43%/42.24% to 48.21%/45.22% for ground-view, and from 61.97%/57.66% to 66.19%/61.87% for drone-view.

cs.CV↗

arXiv2024

Crafting Personalized Agents through Retrieval-Augmented Generation on Editable Memory Graphs

In the age of mobile internet, user data, often referred to as memories, is continuously generated on personal devices. Effectively managing and utilizing this data to deliver services to users is a compelling research topic. In this paper, we introduce a novel task of crafting personalized agents powered by large language models (LLMs), which utilize a user's smartphone memories to enhance downstream applications with advanced LLM capabilities. To achieve this goal, we introduce EMG-RAG, a solution that combines Retrieval-Augmented Generation (RAG) techniques with an Editable Memory Graph (EMG). This approach is further optimized using Reinforcement Learning to address three distinct challenges: data collection, editability, and selectability. Extensive experiments on a real-world dataset validate the effectiveness of EMG-RAG, achieving an improvement of approximately 10% over the best existing approach. Additionally, the personalized agents have been transferred into a real smartphone AI assistant, which leads to enhanced usability.

cs.CL↗

arXiv2024

Hyperbolic site percolation

Several results are presented for site percolation on quasi-transitive, planar graphs $G$ with one end, when properly embedded in either the Euclidean or hyperbolic plane. If $(G_1,G_2)$ is a matching pair derived from some quasi-transitive mosaic $M$, then $p_u(G_1)+p_c(G_2)=1$, where $p_c$ is the critical probability for the existence of an infinite cluster, and $p_u$ is the critical value for the existence of a unique such cluster. This fulfils and extends to the hyperbolic plane an observation of Sykes and Essam(1964), and it extends to quasi-transitive site models a theorem of Benjamini and Schramm (Theorem 3.8, J. Amer. Math. Soc. 14 (2001) 487--507) for transitive bond percolation. It follows that $p_u (G)+p_c (G_*)=p_u(G_*)+p_c(G)=1$, where $G_*$ denotes the matching graph of $G$. In particular, $p_u(G)+p_c(G)\ge 1$ and hence, when $G$ is amenable we have $p_c(G)=p_u(G) \ge \frac12$. When combined with the main result of the companion paper by the same authors ("Percolation critical probabilities of matching lattice-pairs", Random Struct. Alg. 2024), we obtain for transitive $G$ that the strict inequality $p_u(G)+p_c(G)> 1$ holds if and only if $G$ is not a triangulation. A key technique is a method for expressing a planar site percolation process on a matching pair in terms of a dependent bond process on the corresponding dual pair of graphs. Amongst other things, the results reported here answer positively two conjectures of Benjamini and Schramm (Conjectures 7 and 8, Electron. Comm. Probab. 1 (1996) 71--82) in the case of quasi-transitive graphs.

math.PR↗

arXiv2024

Enhancing Complex Causality Extraction via Improved Subtask Interaction and Knowledge Fusion

Event Causality Extraction (ECE) aims at extracting causal event pairs from texts. Despite ChatGPT's recent success, fine-tuning small models remains the best approach for the ECE task. However, existing fine-tuning based ECE methods cannot address all three key challenges in ECE simultaneously: 1) Complex Causality Extraction, where multiple causal-effect pairs occur within a single sentence; 2) Subtask~ Interaction, which involves modeling the mutual dependence between the two subtasks of ECE, i.e., extracting events and identifying the causal relationship between extracted events; and 3) Knowledge Fusion, which requires effectively fusing the knowledge in two modalities, i.e., the expressive pretrained language models and the structured knowledge graphs. In this paper, we propose a unified ECE framework (UniCE to address all three issues in ECE simultaneously. Specifically, we design a subtask interaction mechanism to enable mutual interaction between the two ECE subtasks. Besides, we design a knowledge fusion mechanism to fuse knowledge in the two modalities. Furthermore, we employ separate decoders for each subtask to facilitate complex causality extraction. Experiments on three benchmark datasets demonstrate that our method achieves state-of-the-art performance and outperforms ChatGPT with a margin of at least 30% F1-score. More importantly, our model can also be used to effectively improve the ECE performance of ChatGPT via in-context learning.

cs.CL↗

arXiv2024

Concise and Precise Context Compression for Tool-Using Language Models

Through reading the documentation in the context, tool-using language models can dynamically extend their capability using external tools. The cost is that we have to input lengthy documentation every time the model needs to use the tool, occupying the input window as well as slowing down the decoding process. Given the progress in general-purpose compression, soft context compression is a suitable approach to alleviate the problem. However, when compressing tool documentation, existing methods suffer from the weaknesses of key information loss (specifically, tool/parameter name errors) and difficulty in adjusting the length of compressed sequences based on documentation lengths. To address these problems, we propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. 1) Selective compression strategy mitigates key information loss by deliberately retaining key information as raw text tokens. 2) Block compression strategy involves dividing tool documentation into short chunks and then employing a fixed-length compression model to achieve variable-length compression. This strategy facilitates the flexible adjustment of the compression ratio. Results on API-Bank and APIBench show that our approach reaches a performance comparable to the upper-bound baseline under up to 16x compression ratio.

cs.CL↗

arXiv2024

Perfect Matchings and Essential Spanning Forests in Hyperbolic Double Circle Packings

We investigate perfect matchings and essential spanning forests in planar hyperbolic graphs via circle packings. We prove the existence of nonconstant harmonic Dirichlet functions that vanish in a closed set of the boundary, generalizing a result in \cite{bsinv}. We then prove the existence of extremal infinite volume measures for uniform spanning forests with partially wired boundary conditions and partially free boundary conditions, generalizing a result in \cite{BLPS01}. Using the double circle packing for a pair of dual graphs, we relate the inverse of the weighted adjacency matrix to the difference of Green's functions plus an explicit harmonic Dirichlet function. This gives explicit formulas for the probabilities of any cylindrical events. We prove that the infinite-volume Gibbs measure obtained from approximations by finite domains with exactly two convex white corners converging to two distinct points along the boundary is extremal, yet not invariant with respect to a finite-orbit subgroup of the automorphism group. We then show that under this measure, a.s.~there are no infinite contours in the symmetric difference of two i.i.d.~random perfect matchings. As an application, we prove that the variance of the height difference of two i.i.d.~uniformly weighted perfect matchings under the boundary condition above on a transitive nonamenable planar graph is always finite; in contrast to the 2D uniformly weighted dimer model on a transitive amenable planar graph as proved in \cite{RK01,KOS06}, where the variance of height difference grows in the order of $\log n$, with $n$ being the graph distance to the boundary. This also implies that a.s.~each point is surrounded by finitely many cycles in the symmetric difference of two i.i.d.~perfect matchings, again in contrast to the 2D Euclidean case.

math.PR↗

arXiv2024

Noncentrosymmetric Nowotny Chimney Ladder Ferromagnet Cr4Ge7 with a High Curie Temperature of ~ 207 K

Noncentrosymmetric magnets usually host intriguing magnetic interactions inherent the crystal structure with broken inversion symmetry, which can give rise to rich magnetic behaviors. We report herein the high-pressure synthesis, crystal structure, magnetizations and magnetic structure of a so-called Nowotny chimney ladder compound Cr4Ge7. Our analysis on the powder neutron diffraction data revises the crystal structure as a noncentrosymmetric space group (P-4c2, No.116). It exhibits two magnetic orders within the temperature range of 2 - 400 K. The first order at ~ 207 K associated with a small magnetic moment of ~ 0.75 miuB is assigned to a commensurate ferromagnetic structure with a propagation vector k = (0, 0, 0). The weak itinerant ferromagnet nature should be caused by the complex Cr spin orders from different Wyckoff positions. The second order at ~ 18 K is assumed to arise from a competition between the Dzyaloshinskii-Moria and Heisenberg interactions. The results provide an excellent platform for study on intricate interactions between various magnetic exchanges as well as for the exploration of high temperature exotic magnetic properties which host potential applications in next-generation spintronics.

cond-mat.str-el↗

arXiv2024

Percolation critical probabilities of matching lattice-pairs

A necessary and sufficient condition is established for the strict inequality $p_c(G_*)<p_c(G)$ between the critical probabilities of site percolation on a quasi-transitive, plane graph $G$ and on its matching graph $G_*$. It is assumed that $G$ is properly embedded in either the Euclidean or the hyperbolic plane. When $G$ is transitive, strict inequality holds if and only if $G$ is not a triangulation. The basic approach is the standard method of enhancements, but its implemention has complexity arising from the non-Euclidean (hyperbolic) space, the study of site (rather than bond) percolation, and the generality of the assumption of quasi-transitivity. This result is complementary to the work of the authors ("Hyperbolic site percolation", arXiv:2203.00981) on the equality $p_u(G) + p_c(G_*) = 1$, where $p_u$ is the critical probability for the existence of a unique infinite open cluster. It implies for transitive $G$ that $p_u(G) + p_c(G) \ge 1$, with equality if and only if $G$ is a triangulation.

math.PR↗

arXiv2023

Planar site percolation on semi-transitive graphs

Semi-transitive graphs, defined in \cite{hps98} as examples where ``uniform percolation" holds whenever $p>p_c$, are a large class of graphs more general than quasi-transitive graphs. Let $G$ be a semi-transitive graph with one end which can be properly embedded into the plane with uniformly bounded face degree for finite faces and minimal vertex degree at least 7. We show that $p_u^{site}(G) +p_c^{site}(G_*)=1$, where $G_*$ denotes the matching graph of $G$. This fulfils and extends an observation of Sykes and Essam in 1964 (\cite{SE64}) to semi-transitive graphs.

math.CO↗

arXiv2023

Improving Cross-Task Generalization with Step-by-Step Instructions

Instruction tuning has been shown to be able to improve cross-task generalization of language models. However, it is still challenging for language models to complete the target tasks following the instructions, as the instructions are general and lack intermediate steps. To address this problem, we propose to incorporate the step-by-step instructions to help language models to decompose the tasks, which can provide the detailed and specific procedures for completing the target tasks. The step-by-step instructions are obtained automatically by prompting ChatGPT, which are further combined with the original instructions to tune language models. The extensive experiments on SUP-NATINST show that the high-quality step-by-step instructions can improve cross-task generalization across different model sizes. Moreover, the further analysis indicates the importance of the order of steps of the step-by-step instruction for the improvement. To facilitate future research, we release the step-by-step instructions and their human quality evaluation results.

cs.CL↗