SearcharxivSearch

arXiv subjects

Hernan Picatto

Publications and source records attributed to Hernan Picatto.

2 recordsLinked to original sources

Reconstructing temporal multi-relational firm networks at scale using large language models. The case of the semiconductor industry

The semiconductor industry is foundational to modern technology, yet its complex global multi-relational firm network remains poorly understood, posing challenges to scientists, firms, and policymakers. Traditional analysis relies on proprietary databases that are often expensive, incomplete, and slowly updated, limiting their ability to capture rapidly evolving dependencies. Here, we demonstrate that a novel, generalizable methodology combining Large Language Models (LLMs) with open web data can reconstruct this network and its structural dynamics at scale. We identify and classify supply-chain, partnership, and ownership links from 170 million semiconductor firm webpages, yielding a temporal network of over 1,300 linked firms. We validate link-extraction quality (Precision: 0.884; F1-score: 0.784), network overlap and complementarity with a proprietary database, and consistency with aggregate economic data. Our network reveals a temporary 9% decline in edges during the 2022 chip shortage, rapid increases in the centrality of AI supply-chain bottleneck firms such as NVIDIA, and geographic realignment of interfirm relations amid geopolitical turbulence. This generalizable framework overcomes barriers to transparency and provides essential, up-to-date maps for assessing resilience and informing policy across strategically relevant sectors.

physics.soc-ph

Cost-Effective Big Data Orchestration Using Dagster: A Multi-Platform Approach

The rapid advancement of big data technologies has underscored the need for robust and efficient data processing solutions. Traditional Spark-based Platform-as-a-Service (PaaS) solutions, such as Databricks and Amazon Web Services Elastic MapReduce, provide powerful analytics capabilities but often result in high operational costs and vendor lock-in issues. These platforms, while user-friendly, can lead to significant inefficiencies due to their cost structures and lack of transparent pricing. This paper introduces a cost-effective and flexible orchestration framework using Dagster. Our solution aims to reduce dependency on any single PaaS provider by integrating various Spark execution environments. We demonstrate how Dagster's orchestration capabilities can enhance data processing efficiency, enforce best coding practices, and significantly reduce operational costs. In our implementation, we achieved a 12% performance improvement over EMR and a 40% cost reduction compared to DBR, translating to over 300 euros saved per pipeline run. Our goal is to provide a flexible, developer-controlled computing environment that maintains or improves performance and scalability while mitigating the risks associated with vendor lock-in. The proposed framework supports rapid prototyping and testing, which is essential for continuous development and operational efficiency, contributing to a more sustainable model of large data processing.

cs.DC