| Title | Using a Robust Metadata Management System to Accelerate Scientific Discovery at Extreme Scales |
| Publication Type | Conference Paper |
| Year of Publication | 2018 |
| Authors | Lawson, M., Lofstead, J. |
| Conference Name | 2018 IEEE/ACM 3rd International Workshop on Parallel Data Storage Data Intensive Scalable Computing Systems (PDSW-DISCS) |
| Keywords | -ATDM, -data-tagging, -Decaf, -descriptive-metadata, -EMPRESS, -SIRIUS, Acceleration, Analytical models, atomic operations, composability, Computational modeling, custom-metadata, EMPRESS 1.0, EMPRESS 2.0, Fault tolerance, Fault tolerant systems, meta data, metadata, Metadata Discovery Problem, metadata management system, Production systems, pubcrawl, query functionality, query processing, Resiliency, Scalability, scale-up scientific applications, scientific discovery, software fault tolerance, storage management |
| Abstract | Our previous work, which can be referred to as EMPRESS 1.0, showed that rich metadata management provides a relatively low-overhead approach to facilitating insight from scale-up scientific applications. However, this system did not provide the functionality needed for a viable production system or address whether such a system could scale. Therefore, we have extended our previous work to create EMPRESS 2.0, which incorporates the features required for a useful production system. Through a discussion of EMPRESS 2.0, this paper explores how to incorporate rich query functionality, fault tolerance, and atomic operations into a scalable, storage system independent metadata management system that is easy to use. This paper demonstrates that such a system offers significant performance advantages over HDF5, providing metadata querying that is 150X to 650X faster, and can greatly accelerate post-processing. Finally, since the current implementation of EMPRESS 2.0 relies on an RDBMS, this paper demonstrates that an RDBMS is a viable technology for managing data-oriented metadata. |
| DOI | 10.1109/PDSW-DISCS.2018.00004 |
| Citation Key | lawson_using_2018 |
Using a Robust Metadata Management System to Accelerate Scientific Discovery at Extreme Scales