SOLAR: Scalable Distributed Spatial Joins through Learning-based Optimization
The proliferation of location-based services has led to massive spatial data generation. Spatial join is a crucial database operation that identifies pairs of objects from two spatial datasets based on spatial relationships. Due to the intensive computational demands, spatial joins are often executed in a distributed manner across clusters. However, current systems fail to recognize similarities in the partitioning of spatial data, leading to redundant computations and increased overhead. Recently, incorporating machine learning optimizations into database operations has enhanced efficiency in traditional joins by predicting optimal strategies. However, applying these optimizations to spatial joins poses challenges due to the complex nature of spatial relationships and the variability of spatial data. This paper introduces SOLAR, scalable distributed spatial joins through learning-based optimization. SOLAR operates through offline and online phases. In the offline phase, it learns balanced spatial partitioning based on the similarities between datasets in query workloads seen so far. In the online phase, when a new join query is received, SOLAR evaluates the similarity between the datasets in the new query and the already-seen workloads using the trained learning model. Then, it decides to either reuse an existing partitioner, avoiding unnecessary computational overhead, or partition from scratch. Our extensive experimental evaluation on real-world datasets demonstrates that SOLAR achieves up to 3.6X speedup in overall join runtime and 2.71X speedup in partitioning time compared to state-of-the-art systems.