Federico D. Sacerdoti

arXiv2013

The Software Garden

This paper describes a practical method of developing custom HPC software products using a store of libraries and tools independent from the OS called a "garden". All dependencies from the product to libraries of the underlying OS distribution are carefully severed, isolating the package from instability due to system upgrades and ensuring repeatable deterministic builds on different flavors of Linux. The method also guarantees multiple versions of a software product may exist together and function correctly, greatly facilitating upgrade and rollback. The method is the first known system to expose all past software versions to the designer, and support deterministic single-package rollback without affecting other installed software. An application of this method for building a high performance trading system in C++ is presented.

cs.SE↗

arXiv2010

Performance and Fault Tolerance in the StoreTorrent Parallel Filesystem

With a goal of supporting the timely and cost-effective analysis of Terabyte datasets on commodity components, we present and evaluate StoreTorrent, a simple distributed filesystem with integrated fault tolerance for efficient handling of small data records. Our contributions include an application-OS pipelining technique and metadata structure to increase small write and read performance by a factor of 1-10, and the use of peer-to-peer communication of replica-location indexes to avoid transferring data during parallel analysis even in a degraded state. We evaluated StoreTorrent, PVFS, and Gluster filesystems using 70 storage nodes and 560 parallel clients on an 8-core/node Ethernet cluster with directly attached SATA disks. StoreTorrent performed parallel small writes at an aggregate rate of 1.69 GB/s, and supported reads over the network at 8.47 GB/s. We ported a parallel analysis task and demonstrate it achieved parallel reads at the full aggregate speed of the storage node local filesystems.

cs.DC↗