Abstract
In time-domain astronomy, STLF (Short-Timescale and Large Field-of-view) sky survey is the latest way of sky observation. Compared to traditional sky survey who can only find astronomical phenomena, STLF sky survey can even reveal how short astronomical phenomena evolve. The difference does not only lead the new survey data but also the new analysis style. It requires that database behind STLF sky survey should support continuous analysis on data streaming, real-time analysis on short-term data and complex analysis on long-term historical data. In addition, both insertion and query latencies have strict requirements to ensure that scientific phenomena can be discovered. However, the existing databases cannot support our scenario. In this paper, we propose AstroServ, a distributed system for analysis and management of large-scale and full life-cycle astronomical data. AstroServ’s core components include three data service layers and a query engine. Each data service layer serves for a specific time period of data and query engine can provide the uniform analysis interface on different data. In addition, we also provide many applications including interactive analysis interface and data mining tool to help scientists efficiently use data. The experimental results show that AstroServ can meet the strict performance requirements and the good recognition accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
GWAC will eventually expand to 36 cameras.
References
Apache cassandra. https://cassandra.apache.org/
Apache hbase. http://hbase.apache.org/
Apache kafka. http://kafka.apache.org/
Apache spark. http://spark.apache.org/
Redis. https://redis.io/
Sciserver. http://www.sciserver.org/
The skymapper transient survey. https://arxiv.org/abs/1702.05585
Becla, J., Lim, K.-T., Monkewitz, S., Nieto-Santisteban, M., Thakar, A.: Organizing the LSST database for real-time astronomical processing. In: Astronomical Data Analysis Software and Systems, pp. 114–117 (2008)
Feng, T., Du, Z., Sun, Y., Wei, J., Bi, J., Liu, J.: Real-time anomaly detection of short-time-scale GWAC survey light curves. In: Proceedings of IEEE International Congress on Big Data, pp. 224–231 (2017)
Frieman, J.A., et al.: The sloan digital sky survey - ii: supernova survey: technical summary. Astron. J. 135(1), 338–347 (2008)
Gorski, K.M., Wandelt, B.D., Hansen, F.K., Hivon, E., Banday, A.J.: The healpix primer. Physics (1999)
Huber, M., Chambers, K.C., Flewelling, H., Smartt, S.J., Smith, K., Wright, D.: The pan-starrs survey for transients (PSST). In: IAU General Assembly, vol. 22 (2015)
Idreos, S., Groffen, F., Nes, N., Manegold, S., Mullender, S., Kersten, M., et al.: MonetDB: Two decades of research in column-oriented database architectures. Q. Bull. IEEE Comput. Soc. Techn. Comm. Database Eng. 35(1), 40–45 (2012)
Li, H., Ghodsi, A., Zaharia, M., Shenker, S., Stoica, I.: Tachyon: reliable, memory speed storage for cluster computing frameworks. In: Proceedings of the ACM Symposium on Cloud Computing, pp. 1–15 (2014)
Nieto-Santisteban, M.A., Thakar, A.R., Szalay, A.S.: Cross-matching very large datasets. In: National Science and Technology Council (NSTC) NASA Conference (2007)
Nugent, P., Cao, Y., Kasliwal, M.: The palomar transient factory. In: Proceedings of SPIE, pp. 939702–939702 (2015)
Szalay, A.S., et al.: The SDSS skyserver: public access to the sloan digital sky server data. In: Proceedings of ACM SIGMOD International Conference on Management of Data, pp. 570–581 (2002)
Wan, M., et al.: Column store for GWAC: a high-cadence, high-density, large-scale astronomical light curve pipeline and distributed shared-nothing database. Publ. Astron. Soc. Pac. 128(969), 114501–114532 (2016)
Wang, D.L., Monkewitz, S.M., Lim, K.T., Becla, J.: Qserv: a distributed shared-nothing database for the LSST catalog. In: High Performance Computing, Networking, Storage and Analysis, pp. 1–11 (2011)
Yang, X., et al.: A fast cross-identification algorithm for searching optical transient sources. Astron. Res. Technol. 10(3), 273–282 (2013)
Acknowledgement
This research was partially supported by the grants from the National Key Research and Development Program of China (No. 2016YFB1000602, 2016YFB1000603); the Natural Science Foundation of China (No. 91646203, 61532016, 61532010, 61379050, 61762082); the Fundamental Research Funds for the Central Universities, the Research Funds of Renmin University (No. 11XNL010); and the Science and Technology Opening up Cooperation project of Henan Province (172106000077).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, C. et al. (2019). AstroServ: A Distributed Database for Serving Large-Scale Full Life-Cycle Astronomical Data. In: Li, J., Meng, X., Zhang, Y., Cui, W., Du, Z. (eds) Big Scientific Data Management. BigSDM 2018. Lecture Notes in Computer Science(), vol 11473. Springer, Cham. https://doi.org/10.1007/978-3-030-28061-1_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-28061-1_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28060-4
Online ISBN: 978-3-030-28061-1
eBook Packages: Computer ScienceComputer Science (R0)