arXiv subjects

Stathes Hadjiefthymiades

Publications and source records attributed to Stathes Hadjiefthymiades.

4 recordsLinked to original sources

arXiv2026

Design and Implementation of a Serverless MapReduce Framework for Scalable Data Pipelines

Modern logistics systems tend to generate continuous streams of data from sources such as GPS, IoT sensors, and logistics management systems. The aggregation, processing, and analysis of data have become vital for monitoring operations, optimizing efficiency, and responding quickly to decision making tasks. In this paper, an event-driven MapReduce framework for real-time data processing in logistics environments is presented. This system runs on Kubernetes with Knative and utilizes Apache Kafka as the backbone for communication between the components. This platform is composed of five loosely coupled services that receive, process, and aggregate the incoming data in real-time. Redis is used to preserve workflow metadata, while an AWS S3 service provides persistent storage for the framework. The design is inspired by the MapReduce programming model. It integrates Function-as-a-Service (FaaS) principles with distributed processing techniques that allow configurable scaling based on the workload demands and the underlying hardware. Experimental evaluation shows that the system can scale effectively as the input data volume increases while supporting scale-to-zero, on-demand processing.

cs.DC↗

arXiv2023

Stepwise correlation of multivariate IoT event data based on first-order Markov chains

Correlating events in complex and dynamic IoT environments is a challenging task not only because of the amount of available data that needs to be processed but also due to the call for time efficient data processing. In this paper, we discuss the major steps that should be performed in real- or near real-time event management focusing on event detection and event correlation. We investigate the adoption of a univariate change detection algorithm for real-time event detection and we propose a stepwise event correlation scheme based on a first-order Markov model. The proposed theory is applied on the maritime domain and is validated through extensive experimentation with real sensor streams originating from large-scale sensor networks deployed in a maritime fleet of ships.

cs.DC↗

arXiv2020

A Stock Options Metaphor for Content Delivery Networks

The concept of Stock Options is used to address the scarcity of resources, not adequately addressed by the previous tools of our Prediction Mechanism. Using a Predictive Reservation Scheme, network and disk resources are being monitored through well-established techniques (Kernel Regression Estimators) in a given time frame. Next, an Secondary Market mechanism significantly improves the efficiency and robustness of our Predictive Reservation Scheme by allowing the fast exchange of unused (remaining) resources between the Origin Servers (CDN Clients). This exchange can happen, either by implementing socially optimal practices or by allowing automatic electronic auctions at the end of the day or at shorter time intervals. Finally, we further enhance our Prediction Mechanism; Stock Options are obtained and exercised, depending on the lack of resources at the end of day. As a result, Origin Servers may acquire resources (if required) at a normal price. The effectiveness of our mechanism further improves.

cs.NI↗

arXiv2018

Event Correlation and Forecasting over Multivariate Streaming Sensor Data

Event management in sensor networks is a multidisciplinary field involving several steps across the processing chain. In this paper, we discuss the major steps that should be performed in real- or near real-time event handling including event detection, correlation, prediction and filtering. First, we discuss existing univariate and multivariate change detection schemes for the online event detection over sensor data. Next, we propose an online event correlation scheme that intends to unveil the internal dynamics that govern the operation of a system and are responsible for the generation of various types of events. We show that representation of event dependencies can be accommodated within a probabilistic temporal knowledge representation framework that allows the formulation of rules. We also address the important issue of identifying outdated dependencies among events by setting up a time-dependent framework for filtering the extracted rules over time. The proposed theory is applied on the maritime domain and is validated through extensive experimentation with real sensor streams originating from large-scale sensor networks deployed in ships.

cs.DC↗