arXiv subjects

Daniel E. Lucani

Publications and source records attributed to Daniel E. Lucani.

At least 19 recordsLinked to original sources

arXiv2026

MUFFLe: Efficient Model Update Compression via Generalized Deduplication for Federated Learning

Federated learning is well suited to edge environments but is often limited by the uplink cost of transmitting model updates. This Work-in-Progress paper presents MUFFLe, a communication-efficient update compression scheme that integrates generalized deduplication (GD) into the FedAvg pipeline. MUFFLe deduplicates repeated patterns across the update vector, yielding a fixed-rate, variable-count compression scheme. Preliminary experiments on IID MNIST with 20 clients show that MUFFLe reaches the target accuracy of $92.93\%$ with 38~MB cumulative uplink communication, compared with 75~MB for 8-bit quantization, 86~MB for Top-$k$ sparsification, and 310~MB for uncompressed FedAvg. These results demonstrate the feasibility of applying GD to communication-efficient federated learning.

cs.LG↗

arXiv2026

EntroGD: Scalable Generalized Deduplication for Efficient Direct Analytics on Compressed IoT Data

Massive data streams from IoT and cyber-physical systems must be processed under strict bandwidth, latency, and resource constraints. Generalized Deduplication (GD) is a promising lossless compression framework, as it supports random access and direct analytics on compressed data. However, existing GD algorithms exhibit quadratic complexity $\mathcal{O}(nd^{2})$, which limits their scalability for high-dimensional datasets. This paper proposes \textbf{EntroGD}, an entropy-guided GD framework that decouples analytical fidelity from compression efficiency to achieve linear complexity $\mathcal{O}(nd)$. EntroGD adopts a two-stage design, first constructing compact condensed samples to preserve information critical for analytics, and then applying entropy-based bit selection to maximize compression. Experiments on 18 IoT datasets show that EntroGD reduces configuration time by up to $53.5\times$ compared to state-of-the-art GD compressors. Moreover, by enabling analytics with access to only $2.6\%$ of the original data volume, EntroGD accelerates clustering by up to $31.6\times$ with negligible loss in accuracy. Overall, EntroGD provides a scalable and system-efficient solution for direct analytics on compressed IoT data.

cs.DB↗

arXiv2025

Beampattern Synthesis for Discrete Phase RIS in Communication and Sensing Systems

Extensive research on Reconfigurable Intelligent Surfaces (RIS) has primarily focused on optimizing reflective coefficients for passive beamforming in specific target directions. This optimization typically assumes prior knowledge of the target direction, which is unavailable before the target is detected. To enhance direction estimation, it is critical to develop array pattern synthesis techniques that yield a wider beam by maximizing the received power over the entire target area. Although this challenge has been addressed with active antennas, RIS systems pose a unique challenge due to their inherent phase constraints, which can be continuous or discrete. This work addresses this challenge through a novel array pattern synthesis method tailored for discrete phase constraints in RIS. We introduce a penalty method that pushes these constraints to the boundary of the convex hull. Then, the Minorization-Maximization (MM) method is utilized to reformulate the problem into a convex one. Our numerical results show that our algorithm can generate a wide beam pattern comparable to that achievable with per-power constraints, with both the amplitudes and phases being adjustable. We compare our method with a traditional beam sweeping technique, showing a) several orders of magnitude reduction of the MSE of Angle of Arrival (AOA) at low to medium Signal-to-Noise Ratio (SNR)s; and b) $8$~dB SNR reduction to achieve a high probability of detection.

eess.SP↗

arXiv2025

HyRES: A Hybrid Replication and Erasure Coding Approach to Data Storage

Reliability in distributed storage systems has typically focused on the design and deployment of data replication or erasure coding techniques. Although some scenarios have considered the use of replication for hot data and erasure coding for cold data in the same system, each is designed in isolation. We propose HyRES, a hybrid scheme incorporates the best characteristics of each scheme, thus, resulting in additional design flexibility and better potential performance for the system. We show that HyRES generalizes previously proposed hybrid schemes. We characterize the theoretical performance of HyRES as well as that of replication and erasure coding considering the effects of the size of the storage networks. We validate our theoretical results using simulations. These results show that HyRES can yield simultaneously lower storage costs than replication, lower probabilities of file loss than replication and erasure coding with similar worst case performance, and even lower effective repair traffic than replication when considering the network size.

cs.IT↗

arXiv2025

Touch-Augmented Gaussian Splatting for Enhanced 3D Scene Reconstruction

This paper presents a multimodal framework that integrates touch signals (contact points and surface normals) into 3D Gaussian Splatting (3DGS). Our approach enhances scene reconstruction, particularly under challenging conditions like low lighting, limited camera viewpoints, and occlusions. Different from the visual-only method, the proposed approach incorporates spatially selective touch measurements to refine both the geometry and appearance of the 3D Gaussian representation. To guide the touch exploration, we introduce a two-stage sampling scheme that initially probes sparse regions and then concentrates on high-uncertainty boundaries identified from the reconstructed mesh. A geometric loss is proposed to ensure surface smoothness, resulting in improved geometry. Experimental results across diverse scenarios show consistent improvements in geometric accuracy. In the most challenging case with severe occlusion, the Chamfer Distance is reduced by over 15x, demonstrating the effectiveness of integrating touch cues into 3D Gaussian Splatting. Furthermore, our approach maintains a fully online pipeline, underscoring its feasibility in visually degraded environments.

eess.SP↗

arXiv2025

dreaMLearning: Data Compression Assisted Machine Learning

Despite rapid advancements, machine learning, particularly deep learning, is hindered by the need for large amounts of labeled data to learn meaningful patterns without overfitting and immense demands for computation and storage, which motivate research into architectures that can achieve good performance with fewer resources. This paper introduces dreaMLearning, a novel framework that enables learning from compressed data without decompression, built upon Entropy-based Generalized Deduplication (EntroGeDe), an entropy-driven lossless compression method that consolidates information into a compact set of representative samples. DreaMLearning accommodates a wide range of data types, tasks, and model architectures. Extensive experiments on regression and classification tasks with tabular and image data demonstrate that dreaMLearning accelerates training by up to 8.8x, reduces memory usage by 10x, and cuts storage by 42%, with a minimal impact on model performance. These advancements enhance diverse ML applications, including distributed and federated learning, and tinyML on resource-constrained edge devices, unlocking new possibilities for efficient and scalable learning.

cs.LG↗

arXiv2025

Not all those who drift are lost: Drift correction and calibration scheduling for the IoT

Sensors provide a vital source of data that link digital systems with the physical world. However, as sensors age, the relationship between what they measure and what they output changes. This is known as sensor drift and poses a significant challenge that, combined with limited opportunity for re-calibration, can severely limit data quality over time. Previous approaches to drift correction typically require large volumes of ground truth data and do not consider measurement or prediction uncertainty. In this paper, we propose a probabilistic sensor drift correction method that takes a fundamental approach to modelling the sensor response using Gaussian Process Regression. Tested using dissolved oxygen sensors, our method delivers mean squared error (MSE) reductions of up to 90% and more than 20% on average. We also propose a novel uncertainty-driven calibration schedule optimisation approach that builds on top of drift correction and further reduces MSE by up to 15.7%.

eess.SP↗

arXiv2024

RAGE for the Machine: Image Compression with Low-Cost Random Access for Embedded Applications

We introduce RAGE, an image compression framework that achieves four generally conflicting objectives: 1) good compression for a wide variety of color images, 2) computationally efficient, fast decompression, 3) fast random access of images with pixel-level granularity without the need to decompress the entire image, 4) support for both lossless and lossy compression. To achieve these, we rely on the recent concept of generalized deduplication (GD), which is known to provide efficient lossless (de)compression and fast random access in time-series data, and deliver key expansions suitable for image compression, both lossless and lossy. Using nine different datasets, incl. graphics, logos, natural images, we show that RAGE has similar or better compression ratios to state-of-the-art lossless image compressors, while delivering pixel-level random access capabilities. Tests in an ARM Cortex-M33 platform show seek times between 9.9 and 40.6~ns and average decoding time per pixel between 274 and 1226~ns. Our measurements also show that RAGE's lossy variant, RAGE-Q, outperforms JPEG by several fold in terms of distortion in embedded graphics and has reasonable compression and distortion for natural images.

eess.IV↗

arXiv2024

PairwiseHist: Fast, Accurate and Space-Efficient Approximate Query Processing with Data Compression

Exponential growth in data collection is creating significant challenges for data storage and analytics latency.Approximate Query Processing (AQP) has long been touted as a solution for accelerating analytics on large datasets, however, there is still room for improvement across all key performance criteria. In this paper, we propose a novel histogram-based data synopsis called PairwiseHist that uses recursive hypothesis testing to ensure accurate histograms and can be built on top of data compressed using Generalized Deduplication (GD). We thus show that GD data compression can contribute to AQP. Compared to state-of-the-art AQP approaches, PairwiseHist achieves better performance across all key metrics, including 2.6$ \times $ higher accuracy, 3.5$ \times $ lower latency, 24$ \times $ smaller synopses and 1.5--4$ \times $ faster construction time.

cs.DB↗

arXiv2023

Lossless preprocessing of floating point data to enhance compression

Data compression algorithms typically rely on identifying repeated sequences of symbols from the original data to provide a compact representation of the same information, while maintaining the ability to recover the original data from the compressed sequence. Using data transformations prior to the compression process has the potential to enhance the compression capabilities, being lossless as long as the transformation is invertible. Floating point data presents unique challenges to generate invertible transformations with high compression potential. This paper identifies key conditions for basic operations of floating point data that guarantee lossless transformations. Then, we show four methods that make use of these observations to deliver lossless compression of real datasets, where we improve compression rates up to 40 %.

cs.DB↗

arXiv2023

GreedyGD: Enhanced Generalized Deduplication for Direct Analytics in IoT

Exponential growth in the amount of data generated by the Internet of Things currently pose significant challenges for data communication, storage and analytics and leads to high costs for organisations hoping to leverage their data. Novel techniques are therefore needed to holistically improve the efficiency of data storage and analytics in IoT systems. The emerging compression technique Generalized Deduplication (GD) has been shown to deliver high compression and enable direct compressed data analytics with low storage and memory requirements. In this paper, we propose a new GD-based data compression algorithm called GreedyGD that is designed for analytics. Compared to existing versions of GD, GreedyGD enables more reliable analytics with less data, while running 11.2x faster and delivering even better compression.

cs.DB↗

arXiv2023

Change a Bit to save Bytes: Compression for Floating Point Time-Series Data

The number of IoT devices is expected to continue its dramatic growth in the coming years and, with it, a growth in the amount of data to be transmitted, processed and stored. Compression techniques that support analytics directly on the compressed data could pave the way for systems to scale efficiently to these growing demands. This paper proposes two novel methods for preprocessing a stream of floating point data to improve the compression capabilities of various IoT data compressors. In particular, these techniques are shown to be helpful with recent compressors that allow for random access and analytics while maintaining good compression. Our techniques improve compression with reductions up to 80% when allowing for at most 1% of recovery error.

cs.DS↗

arXiv2022

An Adaptive Column Compression Family for Self-Driving Databases

Modern in-memory databases are typically used for high-performance workloads, therefore they have to be optimized for small memory footprint and high query speed at the same time. Data compression has the potential to reduce memory requirements but often reduces query speed too. In this paper we propose a novel, adaptive compressor that offers a new trade-off point of these dimensions, achieving better compression than LZ4 while reaching query speeds close to the fastest existing segment encoders. We evaluate our compressor both with synthetic data in isolation and on the TPC-H and Join Order Benchmarks, integrated into a modern relational column store, Hyrise.

cs.DB↗

arXiv2022

Bonsai: A Generalized Look at Dual Deduplication

Cloud Service Providers (CSPs) offer a vast amount of storage space at competitive prices to cope with the growing demand for digital data storage. Dual deduplication is a recent framework designed to improve data compression on the CSP while keeping clients' data private from the CSP. To achieve this, clients perform lightweight information-theoretic transformations to their data prior to upload. We investigate the effectiveness of dual deduplication, and propose an improvement for the existing state-of-the-art method. We name our proposal Bonsai as it aims at reducing storage fingerprint and improving scalability. In detail, Bonsai achieves (1) significant reduction in client storage, (2) reduction in total required storage (client + CSP), and (3) reducing the deduplication time on the CSP. Our experiments show that Bonsai achieves compression rates of 68\% on the cloud and 5\% on the client, while allowing the cloud to identify deduplications in a time-efficient manner. We also show that combining our method with universal compressors in the cloud, e.g., Brotli, can yield better overall compression on the data compared to only applying the universal compressor or plain Bonsai. Finally, we show that Bonsai and its variants provide sufficient privacy against an honest-but-curious CPS that knows the distribution of the Clients' original data.

cs.CR↗

arXiv2022

Bifrost: Secure, Scalable and Efficient File Sharing System Using Dual Deduplication

We consider the problem of sharing sensitive or valuable files across users while partially relying on a common, untrusted third-party, e.g., a Cloud Storage Provider (CSP). Although users can rely on a secure peer-to-peer (P2P) channel for file sharing, this introduces potential delay on the data transfer and requires the sender to remain active and connected while the transfer process occurs. Instead of using the P2P channel for the entire file, users can upload information about the file on a common CSP and share only the essential information that enables the receiver to download and recover the original file. This paper introduces Bifrost, an innovative file sharing system inspired by recent results on dual deduplication. Bifrost achieves the desired functionality and simultaneously guarantees that (1) the CSP can efficiently compress outsourced data; (2) the secure P2P channel is used only to transmit short, but crucial information; (3) users can check for data integrity, i.e., detect if the CSP alters the outsourced data; and (4) only the sender (data owner) and the intended receiver can access the file after sharing, i.e., the cloud or no malicious adversary can infer useful information about the shared file. We analyze compression and bandwidth performance using a proof-of-concept implementation. Our experiments show that secure file sharing can be achieved by sending only 650 bits on the P2P channel, irrespective of file size, while the CSP that aids the sharing can enjoy a compression rate of 86.9 %.

cs.CR↗

arXiv2021

Energy Efficient Data Recovery from Corrupted LoRa Frames

High frame-corruption is widely observed in Long Range Wide Area Networks (LoRaWAN) due to the coexistence with other networks in ISM bands and an Aloha-like MAC layer. LoRa's Forward Error Correction (FEC) mechanism is often insufficient to retrieve corrupted data. In fact, real-life measurements show that at least one-fourth of received transmissions are corrupted. When more frames are dropped, LoRa nodes usually switch over to higher spreading factors (SF), thus increasing transmission times and increasing the required energy. This paper introduces ReDCoS, a novel coding technique at the application layer that improves recovery of corrupted LoRa frames, thus reducing the overall transmission time and energy invested by LoRa nodes by several-fold. ReDCoS utilizes lightweight coding techniques to pre-encode the transmitted data. Therefore, the inbuilt Cyclic Redundancy Check (CRC) that follows is computed based on an already encoded data. At the receiver, we use both the CRC and the coded data to recover data from a corrupted frame beyond the built-in Error Correcting Code (ECC). We compare the performance of ReDCoS to (I) the standard FEC of vanilla-LoRaWAN, and to (ii) RS coding applied as ECC to the data of LoRaWAN. The results indicated a 54x and 13.5x improvement of decoding ratio, respectively, when 20 data symbols were sent. Furthermore, we evaluated ReDCoS on-field using LoRa SX1261 transceivers showing that it outperformed RS-coding by factor of at least 2x (and up to 6x) in terms of the decoding ratio while consuming 38.5% less energy per correctly received transmission.

cs.IT↗

arXiv2021

Stream Compression of DLMS Smart Meter Readings

Smart electricity meters typically upload readings a few times a day. Utility providers aim to increase the upload frequency in order to access consumption information in near real time, but the legacy compressors fail to provide sufficient savings on the low-bandwidth, high-cost data connection. We propose a new compression method and data format for DLMS smart meter readings, which is significantly better with frequent uploads and enable reporting every reading in near real time with the same or lower data sizes than the currently available compressors in the DLMS protocol.

cs.NI↗

arXiv2021

QoS-Aware Placement of Deep Learning Services on the Edge with Multiple Service Implementations

Mobile edge computing pushes computationally-intensive services closer to the user to provide reduced delay due to physical proximity. This has led many to consider deploying deep learning models on the edge -- commonly known as edge intelligence (EI). EI services can have many model implementations that provide different QoS. For instance, one model can perform inference faster than another (thus reducing latency) while achieving less accuracy when evaluated. In this paper, we study joint service placement and model scheduling of EI services with the goal to maximize Quality-of-Servcice (QoS) for end users where EI services have multiple implementations to serve user requests, each with varying costs and QoS benefits. We cast the problem as an integer linear program and prove that it is NP-hard. We then prove the objective is equivalent to maximizing a monotone increasing, submodular set function and thus can be solved greedily while maintaining a (1-1/e)-approximation guarantee. We then propose two greedy algorithms: one that theoretically guarantees this approximation and another that empirically matches its performance with greater efficiency. Finally, we thoroughly evaluate the proposed algorithm for making placement and scheduling decisions in both synthetic and real-world scenarios against the optimal solution and some baselines. In the real-world case, we consider real machine learning models using the ImageNet 2012 data-set for requests. Our numerical experiments empirically show that our more efficient greedy algorithm is able to approximate the optimal solution with a 0.904 approximation on average, while the next closest baseline achieves a 0.607 approximation on average.

cs.NI↗