Corbin McElhanney | Searcharxiv

arXiv2026

DySkew: Dynamic Data Redistribution for Skew-Resilient Snowpark UDF Execution

Snowflake revolutionized data warehousing with an elastic architecture that decouples compute and storage, enabling scalable solutions for diverse data analytics needs. Building on this foundation, Snowflake has advanced its AI Data Cloud vision by introducing Snowpark, a managed turnkey solution that supports data engineering and AI/ML workloads using Python and other programming languages. While Snowpark's User-Defined Function (UDF) execution model offers high throughput, it is highly vulnerable to performance degradation from data skew, where uneven data partitioning causes straggler tasks and unpredictable latency. The non-uniform computational cost of arbitrary user code further exacerbates this classic challenge. This paper presents DySkew, a novel, data-skew-aware execution strategy for Snowpark UDFs. Built upon Snowflake's new generalized skew handling solution, an adaptive data distribution mechanism utilizing per-link state machines. DySkew addresses the unique challenges of user-defined logic with goals of fine-grained per-row mitigation, dynamic runtime adaptation, and low-overhead, cost-aware redistribution. Specifically, for Snowpark, we introduce crucial optimizations, including an eager redistribution strategy and a Row Size Model to dynamically manage overhead for extremely large rows. This dynamic approach replaces the limitations of the previous static round-robin method. We detail the architecture of this framework and showcase its effectiveness through performance evaluations and real-world case studies, demonstrating significant improvements in the execution time and resource utilization for large-scale Snowpark UDF workloads.

cs.DC↗

arXiv2023

Quantum dynamics of a fully-blockaded Rydberg atom ensemble

Classical simulation of quantum systems plays an important role in the study of many-body phenomena and in the benchmarking and verification of quantum technologies. Exact simulation is often limited to small systems because the dimension of the Hilbert space increases exponentially with the size of the system. For systems that possess a high degree of symmetry, however, classical simulation can reach much larger sizes. Here, we consider an ensemble of strongly interacting atoms with permutation symmetry, enabling the simulation of dynamics of hundreds of atoms at arbitrarily long evolution times. The system is realized by an ensemble of three-level atoms, where one of the levels corresponds to a highly excited Rydberg state. In the limit of all-to-all Rydberg blockade, the Hamiltonian is invariant under permutation of the atoms. Using techniques from representation theory, we construct a block-diagonal form of the Hamiltonian, where the size of the largest block increases only linearly with the system size. We apply this formalism to derive efficient pulse sequences to prepare arbitrary permutation-invariant quantum states. Moreover, we study the quantum dynamics following a quench, uncovering a parameter regime in which the system thermalizes slowly and exhibits pronounced revivals. Our results create new opportunities for the experimental and theoretical study of large interacting and nonintegrable quantum systems.

quant-ph↗