Advertisement

T-Brain: A Collaboration Platform for Data Scientists

  • Chao-Chun Yeh
  • Sheng-An Chang
  • Yi-Chin Chu
  • Xuan-Yi Lin
  • Yichiao Sun
  • Jiazheng Zhou
  • Shih-Kun Huang
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 733)

Abstract

When data were generated easily and rapidly with mobile services and computing power can increase on demand with the cloud computation service, data scientists who work with huge data can solve challenging problems. Smart intelligent applications such as Go, healthcare and self-driving vehicles show great improvement recently. In addition to those problems, there are still more complex problem such as weather impacts analysis, financial crisis prediction and crime prevention and so on. To overcome those challenging problems, many crossdisciplinarity or interdisciplinary experts have to collaborate for the solutions. In the paper, we propose a collaboration platform and a system design for data scientists to share data, write analytic scripts and discuss topics related with those problems. In current status, eleven dataset were collect ed such as spam mail, malware data, honeynet log, Hadoop workload log and some other open data and based on those dataset and improvement local cache design (i.e., average response time improvement 92.36% and request availability improvement 70%). With the platform, many education and competition activities can be hold successfully on the collaboration platform.

Keywords

Collaboration platform Data scientist Docker Jupyter 

Notes

Acknowledgements

This work was supported in part by Trend Micro, National Center for High-Performance Computing, the Institute for Information Industry under the grant 106-EC-17-D-11-1502 and G367JA1210, Taiwan Information Security Center (TWISC), Academia Sinica, and the Ministry of Science and Technology, Taiwan under the grant 105-2221-E-009-122, and 106-3114-E-009-007.

References

  1. 1.
    Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., et al.: Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016)CrossRefGoogle Scholar
  2. 2.
    Sadek, I., Elawady, M., Shabayek, A.E.R.: Automatic classification of bright retinal lesions via deep network features. arXiv preprint arXiv:1707.02022 (2017)
  3. 3.
    Huval, B., Wang, T., Tandon, S., Kiske, J., Song, W., Pazhayampallil, J., et al.: An empirical evaluation of deep learning on highway driving. arXiv preprint arXiv:1504.01716 (2015)
  4. 4.
    InnoCentive|Open Innovation & Crowdsourcing Platform. https://www.innocentive.com/
  5. 5.
    Kaggle: Your home for data science. https://www.kaggle.com/. Accessed 1 Dec 2017
  6. 6.
    CrowdANALYTIX: Automating business processes using artificial intelligenceGoogle Scholar
  7. 7.
    CodaLab - Home. https://worksheets.codalab.org/. Accessed 1 Dec 2017
  8. 8.
    Topcoder|Deliver Faster through Crowdsourcing. https://www.topcoder.com/
  9. 9.
    HackerRank|Technical Recruiting|Hiring the Best Engineers. https://www.hackerrank.com/. Accessed 1 Dec 2017
  10. 10.
    Merkel, D.: Docker: lightweight Linux containers for consistent development and deployment. Linux J. 2014, 2 (2014)Google Scholar
  11. 11.
    Docker Swarm overview|Docker Documentation. https://docs.docker.com/swarm/overview/. Accessed 1 Dec 2017
  12. 12.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51, 107–113 (2008)CrossRefGoogle Scholar
  13. 13.
    Project Jupyter|Home. http://jupyter.org/. Accessed 1 Dec 2017
  14. 14.
    Kluyver, T., Ragan-Kelley, B., Pérez, F., Granger, B.E., Bussonnier, M., Frederic, J., et al.: Jupyter Notebooks-a publishing format for reproducible computational workflows. In: ELPUB, pp. 87–90 (2016)Google Scholar
  15. 15.
    GitHub - jupyterhub/jupyterhub: Multi-user server for Jupyter notebooks. https://github.com/jupyterhub/jupyterhub. Accessed 1 Dec 2017
  16. 16.
    Overview of RESTful API Description Languages - Wikipedia. https://en.wikipedia.org/wiki/Overview_of_RESTful_API_Description_Languages. Accessed 1 Dec 2017
  17. 17.
    Deploying JupyterHub for Education. https://developer.rackspace.com/blog/deploying-jupyterhub-for-education/. Accessed 1 Dec 2017
  18. 18.
    Redis. https://redis.io. Accessed 1 Dec 2017
  19. 19.
    Celery: Distributed task queue. http://www.celeryproject.org
  20. 20.
    Selenium - Web Browser Automation. http://www.seleniumhq.org
  21. 21.
    keras-mnist-tutorial/MNIST in Keras.ipynb at master · wxs/keras-mnist-tutorial · GitHub. https://github.com/wxs/keras-mnist-tutorial/blob/master/MNIST%20in%20Keras.ipynb
  22. 22.
    OpenChorus Project: The Dawn of The Data Science Movement|Dell EMC Big Data. http://bigdatablog.emc.com/2012/11/09/openchorus-project-the-dawn-of-the-data-science-movement/
  23. 23.
    Zeilenga, K.: Lightweight directory access protocol (LDAP): technical specification road map (2006)Google Scholar
  24. 24.
    Pierson, N.: Overview of Active Directory Federation Services in Windows Server 2003 R2. Microsoft Corporation, October 2005Google Scholar
  25. 25.
    Hellerstein, J.M., Ré, C., Schoppmann, F., Wang, D.Z., Fratkin, E., Gorajek, A., et al.: The MADlib analytics library: or MAD skills, the SQL. Proc. VLDB Endow. 5, 1700–1711 (2012)CrossRefGoogle Scholar
  26. 26.
    RC Team: R language definition. R Foundation for Statistical Computing, Vienna, Austria (2000)Google Scholar
  27. 27.
    Trusted Analytics. https://github.com/trustedanalytics. Accessed 1 Dec 2017
  28. 28.
    Kafka, A.: A high-throughput, distributed messaging system, vol. 5 (2014). kafka.apache.org
  29. 29.
    Zawodny, J.: Redis: lightweight key/value store that goes the extra mile. Linux Magazine, vol. 79 (2009)Google Scholar
  30. 30.
    Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, p. 2 (2012)Google Scholar
  31. 31.
    Vora, M.N.: Hadoop-HBase for large-scale data. In: 2011 International Conference on Computer Science and Network Technology (ICCSNT), pp. 601–605 (2011)Google Scholar
  32. 32.
    Chodorow, K.: MongoDB: The Definitive Guide: Powerful and Scalable Data Storage. O’Reilly Media, Inc. (2013)Google Scholar
  33. 33.
    H2O.ai. https://www.h2o.ai. Accessed 1 Dec 2017
  34. 34.
    RStudio - Open source and enterprise-ready professional software for R. https://www.rstudio.com
  35. 35.
    Pérez, F., Granger, B.E.: IPython: a system for interactive scientific computing. Comput. Sci. Eng. 9 (2007)Google Scholar
  36. 36.
  37. 37.
    Yeh, C.-C., Zhou, J., Chang, S.-A., Lin, X.-Y., Sun, Y., Huang, S.-K.: BigExplorer: a configuration recommendation system for big data platform. In: 2016 Conference on Technologies and Applications of Artificial Intelligence (TAAI), pp. 228–234 (2016)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Chao-Chun Yeh
    • 1
    • 3
  • Sheng-An Chang
    • 3
  • Yi-Chin Chu
    • 3
  • Xuan-Yi Lin
    • 3
  • Yichiao Sun
    • 3
  • Jiazheng Zhou
    • 3
  • Shih-Kun Huang
    • 1
    • 2
  1. 1.Department of Computer ScienceNational Chiao Tung UniversityHsinchuTaiwan
  2. 2.Information Technology Service CenterNational Chiao Tung UniversityHsinchuTaiwan
  3. 3.Computational Intelligence Technology Center, Industrial Technology Research InstituteNational Chiao Tung UniversityHsinchuTaiwan

Personalised recommendations