TY - GEN
T1 - A Machine Learning Approach for Data Quality Control of Earth Observation Data Management System
AU - Han, Weiguo
AU - Jochum, Matthew
N1 - Publisher Copyright:
© 2020 IEEE.
PY - 2020/9/26
Y1 - 2020/9/26
N2 - In the big data era, innovative technologies like cloud computing, artificial intelligence, and machine learning are increasingly utilized in the large-scale data management systems of many industry sectors to make them more scalable and intelligent. Applying them to automate and optimize earth observation data management is a hot topic. To improve data quality control mechanisms, a machine learning method in combination with built-in quality rules is presented in this paper to evolve processes around data quality and enhance management of earth observation data. The rules of quality check are set up to detect the common issues, including data completeness, data latency, bad data, and data duplication, and the machine learning model is trained, tested, and deployed to address these quality issues automatically and reduce manual efforts.
AB - In the big data era, innovative technologies like cloud computing, artificial intelligence, and machine learning are increasingly utilized in the large-scale data management systems of many industry sectors to make them more scalable and intelligent. Applying them to automate and optimize earth observation data management is a hot topic. To improve data quality control mechanisms, a machine learning method in combination with built-in quality rules is presented in this paper to evolve processes around data quality and enhance management of earth observation data. The rules of quality check are set up to detect the common issues, including data completeness, data latency, bad data, and data duplication, and the machine learning model is trained, tested, and deployed to address these quality issues automatically and reduce manual efforts.
KW - Big Data
KW - Data Quality
KW - Data management
KW - Earth Observation Data
KW - Machine Learning
KW - Random Forest
UR - https://www.scopus.com/pages/publications/85102003873
U2 - 10.1109/IGARSS39084.2020.9323615
DO - 10.1109/IGARSS39084.2020.9323615
M3 - Conference contribution
AN - SCOPUS:85102003873
T3 - International Geoscience and Remote Sensing Symposium (IGARSS)
SP - 3101
EP - 3103
BT - 2020 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2020 - Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2020 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2020
Y2 - 26 September 2020 through 2 October 2020
ER -