Breaking the Barriers of Database-Agnostic Transactions
Federated transaction management has long been used as a method to virtually integrate multiple databases from a transactional perspective, ensuring consistency across the databases. Modern approaches manage transactions on top of a database abstraction to achieve database agnosticism; however, these approaches face several challenges. First, managing transactions on top of a database abstraction makes performance optimization difficult because the abstraction hides away the details of underlying databases, such as database-specific capabilities. Additionally, it requires that application data and the associated transaction metadata be colocated in the same record to allow for efficient updates, necessitating a schema migration to run federated transactions on top of existing databases. This paper introduces a new concept in such database abstraction called Atomicity Unit (AU) to address these challenges. AU enables federated transaction management to aggressively pushdown database operations by making use of the knowledge about the scope within which they can perform operations atomically, fully harnessing the performance of the databases. Moreover, AU enables efficient separation of transaction metadata from application data, allowing federated transactions to run on existing databases without requiring a schema migration or significant performance degradation. In this paper, we describe AU, how AU addresses the challenges, and its implementation within ScalarDB, an open-sourced database-agnostic federated transaction manager. We also present evaluation results demonstrating that ScalarDB with AU achieves significantly better performance and efficient metadata separation.