arXiv subjects

Filip Blagojevic

Publications and source records attributed to Filip Blagojevic.

4 recordsLinked to original sources

arXiv2025

AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache

Large Language Models (LLMs) are widely used in generative applications such as chatting, code generation, and reasoning. However, many realworld workloads such as classification, question answering, recommendation, and text embedding rely solely on the prefill stage of inference, where the model encodes input sequences without performing autoregressive decoding. In these prefill only scenarios, the self-attention computation becomes the primary performance bottleneck due to its quadratic complexity with respect to sequence length. In this paper, we observe that semantically different sentences often produce similar attention maps across layers and heads. Building on this insight, we propose AttnCache, a framework that accelerates the prefill stage of LLM inference by retrieving and reusing similar attention maps. Based on an attention map memorization database, AttnCache employs efficient caching and similarity search techniques to identify and reuse pre-cached attention maps during inference, thereby reducing the computational overhead of self-attention. Experimental results show that AttnCache achieves an average of 1.2x end-to-end and 2x attention speedup on CPU, and 1.6x end-to-end and 3x attention speedup on GPU, with negligible accuracy degradation.

cs.CL↗

arXiv2023

AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems

Transformer models gain popularity because of their superior inference accuracy and inference throughput. However, the transformer is computation-intensive, causing a long inference time. The existing works on transformer inference acceleration have limitations caused by either the modification of transformer architectures or the need of specialized hardware. In this paper, we identify the opportunities of using memoization to accelerate the self-attention mechanism in transformers without the above limitations. Built upon a unique observation that there is rich similarity in attention computation across inference sequences, we build a memoization database that leverages the emerging big memory system. We introduce a novel embedding technique to find semantically similar inputs to identify computation similarity. We also introduce a series of techniques such as memory mapping and selective memoization to avoid memory copy and unnecessary overhead. We enable 22% inference-latency reduction on average (up to 68%) with negligible loss in inference accuracy.

cs.PF↗

arXiv2018

Storage-Efficient Shared Memory Emulation

We study the design of storage-efficient algorithms for emulating atomic shared memory over an asynchronous, distributed message-passing system. Our first algorithm is an atomic single-writer multi-reader algorithm based on a novel erasure-coding technique, termed \emph{multi-version code}. Next, we propose an extension of our single-writer algorithm to a multi-writer multi-reader environment. Our second algorithm combines replication and multi-version code, and is suitable in situations where we expect a large number of concurrent writes. Moreover, when the number of concurrent writes is bounded, we propose a simplified variant of the second algorithm that has a simple structure similar to the single-writer algorithm. Let $N$ be the number of servers, and the shared memory variable be of size 1 unit. Our algorithms have the following properties: (i) The write operation terminates if the number of server failures is bounded by a parameter $f$. The algorithms also guarantee the termination of the read as long as the number of writes concurrent with the read is smaller than a design parameter $ν$, and the number of server failures is bounded by $f$. (ii) The overall storage size for the first algorithm, and the steady-state storage size for the second algorithm, are all $N/\lceil \frac{N-2f}ν \rceil$ units. Moreover, our simplified variant of the second algorithm achieves the worst-case storage cost of $N/\lceil \frac{N-2f}ν \rceil$, asymptotically matching a lower bound by Cadambe et al. for $N \gg f, ν\le f+1$. (iii) The write and read operations only consist of a small number (2 to 3) of communication rounds. (iv) For all algorithms, the server maintains a simple data structure. A server only needs to store the information associated with the latest value it observes, similar to replication-based algorithms.

cs.DC↗

arXiv2013

Repair-Optimal MDS Array Codes over GF(2)

Maximum-distance separable (MDS) array codes with high rate and an optimal repair property were introduced recently. These codes could be applied in distributed storage systems, where they minimize the communication and disk access required for the recovery of failed nodes. However, the encoding and decoding algorithms of the proposed codes use arithmetic over finite fields of order greater than 2, which could result in a complex implementation. In this work, we present a construction of 2-parity MDS array codes, that allow for optimal repair of a failed information node using XOR operations only. The reduction of the field order is achieved by allowing more parity bits to be updated when a single information bit is being changed by the user.

cs.IT↗