arXiv subjects

Shuaiyi Li

Publications and source records attributed to Shuaiyi Li.

12 recordsLinked to original sources

arXiv2026

AEON-z5: A Candidate AGN-driven Outflow Enriching the Circumgalactic Medium at $z\simeq5.23$

The dispersal of chemically enriched gas from galaxies into their surroundings is a key process in galaxy evolution, yet direct observational evidence at z>5 remains scarce. We present AEON-z5, a galaxy at z~5.23 in the COSMOS field comprising a compact continuum-emitting core surrounded by an extended, line-dominated ionized nebula. JWST/NIRCam imaging shows that H$\alpha$+[N II] and H$\beta$+[O III] emission extends to projected radii of >4 kpc - 2-2.5 times the typical effective radius at the host stellar mass - reaching the outer ISM and the inner CGM. F444W grism data reveal a highly asymmetric H$\alpha$+[N II] profile, which we interpret as a bipolar outflow and decompose into three kinematic components. The dominant component has a flux-weighted velocity offset of ~+489 km/s and FWHM~630 km/s, with a wing reaching $v_{84}$~792 km/s. A tentative detached feature at $\Delta v_{LOS}$~2800 km/s, detected at 1.5$\sigma$ in the 1D spectrum (2.4$\sigma$ in the 2D fit), may trace an outflow clump. These kinematics are most coherently explained by an AGN in the core. For the extended nebula, we derive a velocity curve shifting redward with radius, reaching 200-400 km/s at $r_p$~1.3-2.5 kpc - a trend that may reflect an accelerating outflow, corotating gas, or recycled inflow viewed in projection. Crucially, the measured [N II]/H$\alpha$ ratios imply near-solar N2-based abundances (0.7-1.0 $Z_\odot$), remaining >0.37 $Z_\odot$ after allowing for AGN excitation and calibration systematics. The combination of large extent, enrichment, and extreme kinematics identifies AEON-z5 as a candidate snapshot of feedback-driven metal transport, offering a direct view of how early AGN activity may redistribute chemically processed gas into the CGM within the first ~1.1 billion years.

astro-ph.GA↗

arXiv2026

Early Emergence of Environmental Effects: Accelerated Galaxy Assembly in a z=2.96 Protocluster in the COSMOS Field

The redshift range z=2-4 marks a critical transition in large scale structure formation, where the dynamically unrelaxed progenitors of local massive clusters undergo rapid stellar mass assembly. We report the discovery and physical characterization of a highly significant protocluster, PC J1001+0214, at z=2.96 within the COSMOS field. Leveraging the multi-wavelength COSMOS2025 catalog with exceptional photometric precision in conjunction with JWST/NIRCam wide-field slitless spectroscopy (WFSS) from the COSMOS-3D program, we robustly identify a cosmic overdensity with $\delta=2.66$. The structure comprises 131 member galaxies, including 21 spectroscopically confirmed members (identified primarily via He I $\lambda10830$ emission) and 110 high-fidelity photometric members. A comparative analysis against a mass-complete coeval field sample reveals a statistically significant +0.2 dex shift in the stellar mass distribution of protocluster members, signaling accelerated mass assembly in the dense environment by $z\sim3$. While the protocluster population broadly follows the star-forming main sequence, low-to-intermediate mass galaxies ($\log_{10}(M_{*}/M_{\odot})\le9.7$) exhibit a measurable star formation rate (SFR) enhancement of +0.11 to +0.15 dex. Crucially, the quiescent fraction remains extremely low and indistinguishable from the field sample, implying that environmental quenching mechanisms have not yet become dominant. Furthermore, a preliminary size-mass analysis hints at elevated morphological compactness among protocluster members at fixed stellar mass. These results suggest that PC J1001+0214 represents a growth-dominated protocluster phase in which environmental effects are already detectable, primarily through accelerated stellar mass assembly and ongoing growth rather than through strong quenching.

astro-ph.GA↗

arXiv2026

Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation

Block attention, which processes the input as separate blocks that cannot attend to one another, offers significant potential to improve KV cache reuse in long-context scenarios such as Retrieval-Augmented Generation (RAG). However, its broader application is hindered by two key challenges: the difficulty of segmenting input text into meaningful, self-contained blocks, and the inefficiency of existing block fine-tuning methods that risk degrading performance. To address these, we first construct SemanticSeg, a large and diverse semantic segmentation dataset containing over 30k instances across 16 categories-including books, code, web text, and conversations with text lengths ranging from 2k to 32k. Using this dataset, we train a lightweight segmenter to automatically partition text into human-instinct-aligned blocks with controllable granularity. Second, we propose block distillation, a training framework that is more efficient than block fine-tuning, which uses a frozen full-attention teacher model to guide the block-attention student. This framework integrates three novel components: block sink tokens to mitigate information loss at block boundaries, block dropout to leverage training signals from all blocks, and token-level loss weighting to focus learning on block-attention-sensitive tokens. Experiments across multiple models and benchmarks demonstrate that our segmenter outperforms heuristic and statistical baselines, and block distillation achieves near-full-attention performance under block attention, establishing a practical and scalable pathway for deploying block attention.

cs.CL↗

arXiv2025

UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression

Large language models are increasingly capable of handling long-context inputs, but the memory overhead of key-value (KV) cache remains a major bottleneck for general-purpose deployment. While various compression strategies have been explored, sequence-level compression, which drops the full KV caches for certain tokens, is particularly challenging as it can lead to the loss of important contextual information. To address this, we introduce UniGist, a sequence-level long-context compression framework that efficiently preserves context information by replacing raw tokens with special compression tokens (gists) in a fine-grained manner. We adopt a chunk-free training strategy and design an efficient kernel with a gist shift trick, enabling optimized GPU training. Our scheme also supports flexible inference by allowing the actual removal of compressed tokens, resulting in real-time memory savings. Experiments across multiple long-context tasks demonstrate that UniGist significantly improves compression quality, with especially strong performance in detail-recalling tasks and long-range dependency modeling.

cs.CL↗

arXiv2025

InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing

Although existing model editing methods perform well in recalling exact edit facts, they often struggle in complex scenarios that require deeper semantic understanding rather than mere knowledge regurgitation. Leveraging the strong contextual reasoning abilities of large language models (LLMs), in-context learning (ICL) becomes a promising editing method by comprehending edit information through context encoding. However, this method is constrained by the limited context window of LLMs, leading to degraded performance and efficiency as the number of edits increases. To overcome this limitation, we propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts through explicit compression and selection mechanisms. Specifically, InComeS compresses each editing context into the key-value (KV) cache of a special gist token, enabling efficient handling of multiple edits without being restricted by the model's context window. Furthermore, specialized cross-attention modules are added to dynamically select the most relevant information from the gist pools, enabling adaptive and effective utilization of edit information. We conduct experiments on diverse model editing benchmarks with various editing formats, and the results demonstrate the effectiveness and efficiency of our method.

cs.CL↗

arXiv2024

A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression

In this work, we provide a thorough investigation of gist-based context compression methods to improve long-context processing in large language models. We focus on two key questions: (1) How well can these methods replace full attention models? and (2) What potential failure patterns arise due to compression? Through extensive experiments, we show that while gist-based compression can achieve near-lossless performance on tasks like retrieval-augmented generation and long-document QA, it faces challenges in tasks like synthetic recall. Furthermore, we identify three key failure patterns: lost by the boundary, lost if surprise, and lost along the way. To mitigate these issues, we propose two effective strategies: fine-grained autoencoding, which enhances the reconstruction of original token information, and segment-wise token importance estimation, which adjusts optimization based on token dependencies. Our work provides valuable insights into the understanding of gist token-based context compression and offers practical strategies for improving compression capabilities.

cs.CL↗

arXiv2024

Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models

Large language models have shown remarkable performance across a wide range of language tasks, owing to their exceptional capabilities in context modeling. The most commonly used method of context modeling is full self-attention, as seen in standard decoder-only Transformers. Although powerful, this method can be inefficient for long sequences and may overlook inherent input structures. To address these problems, an alternative approach is parallel context encoding, which splits the context into sub-pieces and encodes them parallelly. Because parallel patterns are not encountered during training, naively applying parallel encoding leads to performance degradation. However, the underlying reasons and potential mitigations are unclear. In this work, we provide a detailed analysis of this issue and identify that unusually high attention entropy can be a key factor. Furthermore, we adopt two straightforward methods to reduce attention entropy by incorporating attention sinks and selective mechanisms. Experiments on various tasks reveal that these methods effectively lower irregular attention entropy and narrow performance gaps. We hope this study can illuminate ways to enhance context modeling mechanisms.

cs.CL↗

arXiv2024

Knowledge Boundary of Large Language Models: A Survey

Although large language models (LLMs) store vast amount of knowledge in their parameters, they still have limitations in the memorization and utilization of certain knowledge, leading to undesired behaviors such as generating untruthful and inaccurate responses. This highlights the critical need to understand the knowledge boundary of LLMs, a concept that remains inadequately defined in existing research. In this survey, we propose a comprehensive definition of the LLM knowledge boundary and introduce a formalized taxonomy categorizing knowledge into four distinct types. Using this foundation, we systematically review the field through three key lenses: the motivation for studying LLM knowledge boundaries, methods for identifying these boundaries, and strategies for mitigating the challenges they present. Finally, we discuss open challenges and potential research directions in this area. We aim for this survey to offer the community a comprehensive overview, facilitate access to key issues, and inspire further advancements in LLM knowledge research.

cs.CL↗

arXiv2024

Consecutive Batch Model Editing with HooK Layers

As the typical retraining paradigm is unacceptably time- and resource-consuming, researchers are turning to model editing to find an effective way that supports both consecutive and batch scenarios to edit the model behavior directly. Despite all these practical expectations, existing model editing methods fail to realize all of them. Furthermore, the memory demands for such sequential model editing approaches tend to be prohibitive, frequently necessitating an external memory that grows incrementally over time. To cope with these challenges, we propose CoachHooK, a model editing method that simultaneously supports sequential and batch editing. CoachHooK is memory-friendly as it only needs a small amount of it to store several hook layers whose size remains unchanged over time. Experimental results demonstrate the superiority of our method over other batch-supportive model editing methods under both single-round and consecutive batch editing scenarios. Extensive analyses of CoachHooK have been conducted to verify the stability of our method over a number of consecutive steps.

cs.CL↗

arXiv2024

On the Transformations across Reward Model, Parameter Update, and In-Context Prompt

Despite the general capabilities of pre-trained large language models (LLMs), they still need further adaptation to better serve practical applications. In this paper, we demonstrate the interchangeability of three popular and distinct adaptation tools: parameter updating, reward modeling, and in-context prompting. This interchangeability establishes a triangular framework with six transformation directions, each of which facilitates a variety of applications. Our work offers a holistic view that unifies numerous existing studies and suggests potential research directions. We envision our work as a useful roadmap for future research on LLMs.

cs.CL↗

arXiv2024

WatME: Towards Lossless Watermarking Through Lexical Redundancy

Text watermarking has emerged as a pivotal technique for identifying machine-generated text. However, existing methods often rely on arbitrary vocabulary partitioning during decoding to embed watermarks, which compromises the availability of suitable tokens and significantly degrades the quality of responses. This study assesses the impact of watermarking on different capabilities of large language models (LLMs) from a cognitive science lens. Our finding highlights a significant disparity; knowledge recall and logical reasoning are more adversely affected than language generation. These results suggest a more profound effect of watermarking on LLMs than previously understood. To address these challenges, we introduce Watermarking with Mutual Exclusion (WatME), a novel approach leveraging linguistic prior knowledge of inherent lexical redundancy in LLM vocabularies to seamlessly integrate watermarks. Specifically, WatME dynamically optimizes token usage during the decoding process by applying a mutually exclusive rule to the identified lexical redundancies. This strategy effectively prevents the unavailability of appropriate tokens and preserves the expressive power of LLMs. We provide both theoretical analysis and empirical evidence showing that WatME effectively preserves the diverse capabilities of LLMs while ensuring watermark detectability.

cs.CL↗

arXiv2024

DepWiGNN: A Depth-wise Graph Neural Network for Multi-hop Spatial Reasoning in Text

Spatial reasoning in text plays a crucial role in various real-world applications. Existing approaches for spatial reasoning typically infer spatial relations from pure text, which overlooks the gap between natural language and symbolic structures. Graph neural networks (GNNs) have showcased exceptional proficiency in inducing and aggregating symbolic structures. However, classical GNNs face challenges in handling multi-hop spatial reasoning due to the over-smoothing issue, i.e., the performance decreases substantially as the number of graph layers increases. To cope with these challenges, we propose a novel Depth-Wise Graph Neural Network (DepWiGNN). Specifically, we design a novel node memory scheme and aggregate the information over the depth dimension instead of the breadth dimension of the graph, which empowers the ability to collect long dependencies without stacking multiple layers. Experimental results on two challenging multi-hop spatial reasoning datasets show that DepWiGNN outperforms existing spatial reasoning methods. The comparisons with the other three GNNs further demonstrate its superiority in capturing long dependency in the graph.

cs.CL↗