arXiv subjects

Daniel de Oliveira

Publications and source records attributed to Daniel de Oliveira.

12 recordsLinked to original sources

arXiv2026

Autark: A Serverless Toolkit for Prototyping Urban Visual Analytics Systems

The development of visual analytics (VA) systems has traditionally been a labor-intensive process, balancing design methodologies with complex software engineering practices. In domain-specific fields like urban VA, this challenge is amplified by heterogeneous data streams and a reliance on complex, multi-service architectures that hinder fast development, deployment, and reproducibility. Despite the richness of the urban VA literature, the field lacks a consolidated toolkit that encapsulates the core components of these systems, such as spatial data management, analytical processing, and visualization, into a unified, lightweight framework. In this paper, we introduce Autark, a serverless toolkit designed for the rapid prototyping of urban VA systems. Autark provides domain-aware abstractions through a self-contained architecture, enabling researchers to transition from design intention to deployed, shareable systems within hours. Furthermore, Autark's structured, tightly scoped interfaces make it well-suited for AI-assisted coding workflows, where LLMs produce more reliable code when composing from well-defined abstractions rather than generating complex solutions from scratch. Our contributions are: (1) the Autark toolkit, a serverless architecture for rapid prototyping of urban VA; (2) a comparative study of LLM coding effectiveness with and without Autark; and (3) a series of usage scenarios demonstrating its capability to streamline the creation of robust, shareable urban VA prototypes. Autark is available at https://autarkjs.org/.

cs.HC↗

arXiv2024

Workflows Community Summit 2024: Future Trends and Challenges in Scientific Workflows

The Workflows Community Summit gathered 111 participants from 18 countries to discuss emerging trends and challenges in scientific workflows, focusing on six key areas: time-sensitive workflows, AI-HPC convergence, multi-facility workflows, heterogeneous HPC environments, user experience, and FAIR computational workflows. The integration of AI and exascale computing has revolutionized scientific workflows, enabling higher-fidelity models and complex, time-sensitive processes, while introducing challenges in managing heterogeneous environments and multi-facility data dependencies. The rise of large language models is driving computational demands to zettaflop scales, necessitating modular, adaptable systems and cloud-service models to optimize resource utilization and ensure reproducibility. Multi-facility workflows present challenges in data movement, curation, and overcoming institutional silos, while diverse hardware architectures require integrating workflow considerations into early system design and developing standardized resource management tools. The summit emphasized improving user experience in workflow systems and ensuring FAIR workflows to enhance collaboration and accelerate scientific discovery. Key recommendations include developing standardized metrics for time-sensitive workflows, creating frameworks for cloud-HPC integration, implementing distributed-by-design workflow modeling, establishing multi-facility authentication protocols, and accelerating AI integration in HPC workflow management. The summit also called for comprehensive workflow benchmarks, workflow-specific UX principles, and a FAIR workflow maturity model, highlighting the need for continued collaboration in addressing the complex challenges posed by the convergence of AI, HPC, and multi-facility research environments.

cs.DC↗

arXiv2024

Urban Computing for Climate and Environmental Justice: Early Perspectives From Two Research Initiatives

The impacts of climate change are intensifying existing vulnerabilities and disparities within urban communities around the globe, as extreme weather events, including floods and heatwaves, are becoming more frequent and severe, disproportionately affecting low-income and underrepresented groups. Tackling these increasing challenges requires novel approaches that integrate expertise across multiple domains, including computer science, engineering, climate science, and public health. Urban computing can play a pivotal role in these efforts by integrating data from multiple sources to support decision-making and provide actionable insights into weather patterns, infrastructure weaknesses, and population vulnerabilities. However, the capacity to leverage technological advancements varies significantly between the Global South and Global North. In this paper, we present two multiyear, multidisciplinary projects situated in Chicago, USA and Niterói, Brazil, highlighting the opportunities and limitations of urban computing in these diverse contexts. Reflecting on our experiences, we then discuss the essential requirements, as well as existing gaps, for visual analytics tools that facilitate the understanding and mitigation of climate-related risks in urban environments.

cs.CY↗

arXiv2024

Curio: A Dataflow-Based Framework for Collaborative Urban Visual Analytics

Over the past decade, several urban visual analytics systems and tools have been proposed to tackle a host of challenges faced by cities, in areas as diverse as transportation, weather, and real estate. Many of these tools have been designed through collaborations with urban experts, aiming to distill intricate urban analysis workflows into interactive visualizations and interfaces. However, the design, implementation, and practical use of these tools still rely on siloed approaches, resulting in bespoke applications that are difficult to reproduce and extend. At the design level, these tools undervalue rich data workflows from urban experts, typically treating them only as data providers and evaluators. At the implementation level, they lack interoperability with other technical frameworks. At the practical use level, they tend to be narrowly focused on specific fields, inadvertently creating barriers to cross-domain collaboration. To address these gaps, we present Curio, a framework for collaborative urban visual analytics. Curio uses a dataflow model with multiple abstraction levels (code, grammar, GUI elements) to facilitate collaboration across the design and implementation of visual analytics components. The framework allows experts to intertwine data preprocessing, management, and visualization stages while tracking the provenance of code and visualizations. In collaboration with urban experts, we evaluate Curio through a diverse set of usage scenarios targeting urban accessibility, urban microclimate, and sunlight access. These scenarios use different types of data and domain methodologies to illustrate Curio's flexibility in tackling pressing societal challenges. Curio is available at https://urbantk.org/curio.

cs.HC↗

arXiv2024

ProvDeploy: Provenance-oriented Containerization of High Performance Computing Scientific Workflows

Many existing scientific workflows require High Performance Computing environments to produce results in a timely manner. These workflows have several software library components and use different environments, making the deployment and execution of the software stack not trivial. This complexity increases if the user needs to add provenance data capture services to the workflow. This manuscript introduces ProvDeploy to assist the user in configuring containers for scientific workflows with integrated provenance data capture. ProvDeploy was evaluated with a Scientific Machine Learning workflow, exploring containerization strategies focused on provenance in two distinct HPC environments

cs.DC↗

arXiv2023

ProWis: A Visual Approach for Building, Managing, and Analyzing Weather Simulation Ensembles at Runtime

Weather forecasting is essential for decision-making and is usually performed using numerical modeling. Numerical weather models, in turn, are complex tools that require specialized training and laborious setup and are challenging even for weather experts. Moreover, weather simulations are data-intensive computations and may take hours to days to complete. When the simulation is finished, the experts face challenges analyzing its outputs, a large mass of spatiotemporal and multivariate data. From the simulation setup to the analysis of results, working with weather simulations involves several manual and error-prone steps. The complexity of the problem increases exponentially when the experts must deal with ensembles of simulations, a frequent task in their daily duties. To tackle these challenges, we propose ProWis: an interactive and provenance-oriented system to help weather experts build, manage, and analyze simulation ensembles at runtime. Our system follows a human-in-the-loop approach to enable the exploration of multiple atmospheric variables and weather scenarios. ProWis was built in close collaboration with weather experts, and we demonstrate its effectiveness by presenting two case studies of rainfall events in Brazil.

cs.HC↗

arXiv2023

Datasets for Portuguese Legal Semantic Textual Similarity: Comparing weak supervision and an annotation process approaches

The Brazilian judiciary has a large workload, resulting in a long time to finish legal proceedings. Brazilian National Council of Justice has established in Resolution 469/2022 formal guidance for document and process digitalization opening up the possibility of using automatic techniques to help with everyday tasks in the legal field, particularly in a large number of texts yielded on the routine of law procedures. Notably, Artificial Intelligence (AI) techniques allow for processing and extracting useful information from textual data, potentially speeding up the process. However, datasets from the legal domain required by several AI techniques are scarce and difficult to obtain as they need labels from experts. To address this challenge, this article contributes with four datasets from the legal domain, two with documents and metadata but unlabeled, and another two labeled with a heuristic aiming at its use in textual semantic similarity tasks. Also, to evaluate the effectiveness of the proposed heuristic label process, this article presents a small ground truth dataset generated from domain expert annotations. The analysis of ground truth labels highlights that semantic analysis of domain text can be challenging even for domain experts. Also, the comparison between ground truth and heuristic labels shows that heuristic labels are useful.

cs.CL↗

arXiv2021

HELIX: Data-driven characterization of Brazilian land snails

Decision-support systems benefit from hidden patterns extracted from digital information. In the specific domain of gastropod characterization, morphometrical measurements support biologists in the identification of land snail specimens. Although snails can be easily identified by their excretory and reproductive systems, the after-death mollusk body is commonly inaccessible because of either soft material deterioration or fossilization. This study aims at characterizing Brazilian land snails by morphometrical data features manually taken from the shells. In particular, we examined a dataset of shells by using different learning models that labeled snail specimens with a precision up to 97.5% (F1-Score = .975, CKC = .967 and ROC Area = .998). The extracted patterns describe similarities and trends among land snail species and indicates possible outliers physiologies due to climate traits and breeding. Finally, we show some morphometrical characteristics dominate others according to different feature selection biases. Those data-based patterns can be applied to fast land snail identification whenever their bodies are unavailable, as in the recurrent cases of lost shells in nature or private and museum collections.

q-bio.QM↗

arXiv2021

Distributed In-memory Data Management for Workflow Executions

Complex scientific experiments from various domains are typically modeled as workflows and executed on large-scale machines using a Parallel Workflow Management System (WMS). Since such executions usually last for hours or days, some WMSs provide user steering support, i.e., they allow users to run data analyses and, depending on the results, adapt the workflows at runtime. A challenge in the parallel execution control design is to manage workflow data for efficient executions while enabling user steering support. Data access for high scalability is typically transaction-oriented, while for data analysis, it is online analytical-oriented so that managing such hybrid workloads makes the challenge even harder. In this work, we present SchalaDB, an architecture with a set of design principles and techniques based on distributed in-memory data management for efficient workflow execution control and user steering. We propose a distributed data design for scalable workflow task scheduling and high availability driven by a parallel and distributed in-memory DBMS. To evaluate our proposal, we develop d-Chiron, a WMS designed according to SchalaDB's principles. We carry out an extensive experimental evaluation on an HPC cluster with up to 960 computing cores. Among other analyses, we show that even when running data analyses for user steering, SchalaDB's overhead is negligible for workloads composed of hundreds of concurrent tasks on shared data. Our results encourage workflow engine developers to follow a parallel and distributed data-oriented approach not only for scheduling and monitoring but also for user steering.

cs.DC↗

arXiv2019

A superpixel-driven deep learning approach for the analysis of dermatological wounds

Background. The image-based identification of distinct tissues within dermatological wounds enhances patients' care since it requires no intrusive evaluations. This manuscript presents an approach, we named QTDU, that combines deep learning models with superpixel-driven segmentation methods for assessing the quality of tissues from dermatological ulcers. Method. QTDU consists of a three-stage pipeline for the obtaining of ulcer segmentation, tissues' labeling, and wounded area quantification. We set up our approach by using a real and annotated set of dermatological ulcers for training several deep learning models to the identification of ulcered superpixels. Results. Empirical evaluations on 179,572 superpixels divided into four classes showed QTDU accurately spot wounded tissues (AUC = 0.986, sensitivity = 0.97, and specificity = 0.974) and outperformed machine-learning approaches in up to 8.2% regarding F1-Score through fine-tuning of a ResNet-based model. Last, but not least, experimental evaluations also showed QTDU correctly quantified wounded tissue areas within a 0.089 Mean Absolute Error ratio. Conclusions. Results indicate QTDU effectiveness for both tissue segmentation and wounded area quantification tasks. When compared to existing machine-learning approaches, the combination of superpixels and deep learning models outperformed the competitors within strong significant levels.

eess.IV↗

arXiv2015

Detecting and Handling Flash-Crowd Events on Cloud Environments

Cloud computing is a highly scalable computing paradigm where resources are delivered to users on demand via Internet. There are several areas that can benefit from cloud computing and one in special is gaining much attention: the flash-crowd handling. Flash-crowd events happen when servers are unable to handle the volume of requests for a specific content (or a set of contents) that actually reach it, thus causing some requests to be denied. For the handling of flash-crowd events in Web applications, clouds can offer elastic computing and storage capacity during these events in order to process all requests. However, it is important that flash-crowd events are quickly detected and the amount of resources to be instantiated during flash crowds is correctly estimated. In this paper, a new mechanism for detection of flash crowds based on concepts of entropy and total correlation is proposed. Moreover, the Flash-Crowd Handling Problem (FCHP) is precisely defined and formulated as an integer programming problem. A new algorithm for solving it, named FCHP-ILS, is also proposed. With FCHP-ILS the Web provider is able to replicate contents in the available resources and define the types and amount of resources to instantiate in the cloud during a flash-crowd event. Finally we present a case study, based on a synthetic dataset representing flash-crowd events in small scenarios aiming at comparing the proposed approach with de facto standard Amazon's Auto Scaling mechanism.

cs.DC↗

arXiv2014

Handling Flash-Crowd Events to Improve the Performance of Web Applications

Cloud computing can offer a set of computing resources according to users' demand. It is suitable to be used to handle flash-crowd events in Web applications due to its elasticity and on-demand characteristics. Thus, when Web applications need more computing or storage capacity, they just instantiate new resources. However, providers have to estimate the amount of resources to instantiate to handle with the flash-crowd event. This estimation is far from trivial since each cloud environment provides several kinds of heterogeneous resources, each one with its own characteristics such as bandwidth, CPU, memory and financial cost. In this paper, the Flash Crowd Handling Problem (FCHP) is precisely defined and formulated as an integer programming problem. A new algorithm for handling with a flash crowd named FCHP-ILS is also proposed. With FCHP-ILS the Web applications can replicate contents in the already instantiated resources and define the types and amount of resources to instantiate in the cloud during a flash crowd. Our approach is evaluated considering real flash crowd traces obtained from the related literature. We also present a case study, based on a synthetic dataset representing flash-crowd events in small scenarios aiming at the comparison of the proposed approach against Amazon's Auto-Scale mechanism.

cs.DC↗