Data Backbone Design

Abstract

The Data Backbone (DBB) is a key component that provides for data storage, transport, and replication, allowing data products to move between enclaves. This service provides policy-based replication of files (including images and flat files to be loaded into databases as well as other raw and intermediate files) and databases (including metadata about files as well as other miscellaneous databases but not including the large Data Release catalogs) across multiple physical locations, including the Base, Commissioning Cluster, NCSA, and Data Access Centers. It manages caches of files at each endpoint as well as persistence to long-term archival storage (e.g. tape). It provides a registration mechanism for new datasets and database entries and a retrieval mechanism compatible with the Data Butler.