Exploring Metadata Search Essentials for Scientific Data Management
Title | Exploring Metadata Search Essentials for Scientific Data Management |
Publication Type | Conference Paper |
Year of Publication | 2019 |
Authors | Zhang, W., Byna, S., Niu, C., Chen, Y. |
Conference Name | 2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC) |
Date Published | Dec. 2019 |
Publisher | IEEE |
ISBN Number | 978-1-7281-4535-8 |
Keywords | astronomy computing, Complexity theory, compositionality, data management, data structure, data structures, HDF5, indexing, indexing data structure, meta data, metadata, metadata attributes, metadata characteristics, Metadata Discovery Problem, Metadata Indexing, metadata indexing methodologies, metadata queries, Metadata Search, pubcrawl, query processing, real-world astronomy observation dataset, resilience, Resiliency, Scalability, scientific data management, scientific file formats, search problems, tree data structures |
Abstract | Scientific experiments and observations store massive amounts of data in various scientific file formats. Metadata, which describes the characteristics of the data, is commonly used to sift through massive datasets in order to locate data of interest to scientists. Several indexing data structures (such as hash tables, trie, self-balancing search trees, sparse array, etc.) have been developed as part of efforts to provide an efficient method for locating target data. However, efficient determination of an indexing data structure remains unclear in the context of scientific data management, due to the lack of investigation on metadata, metadata queries, and corresponding data structures. In this study, we perform a systematic study of the metadata search essentials in the context of scientific data management. We study a real-world astronomy observation dataset and explore the characteristics of the metadata in the dataset. We also study possible metadata queries based on the discovery of the metadata characteristics and evaluate different data structures for various types of metadata attributes. Our evaluation on real-world dataset suggests that trie is a suitable data structure when prefix/suffix query is required, otherwise hash table should be used. We conclude our study with a summary of our findings. These findings provide a guideline and offers insights in developing metadata indexing methodologies for scientific applications. |
URL | https://ieeexplore.ieee.org/document/8990452/ |
DOI | 10.1109/HiPC.2019.00021 |
Citation Key | zhang_exploring_2019 |
- Metadata Indexing
- tree data structures
- search problems
- scientific file formats
- scientific data management
- Scalability
- Resiliency
- resilience
- real-world astronomy observation dataset
- query processing
- pubcrawl
- Metadata Search
- metadata queries
- metadata indexing methodologies
- astronomy computing
- Metadata Discovery Problem
- metadata characteristics
- metadata attributes
- metadata
- meta data
- indexing data structure
- indexing
- HDF5
- data structures
- data structure
- data management
- Compositionality
- Complexity theory