Skip to main content
SpringerLink
Log in

Scraping Data

  • Chapter
  • First Online:
Data Wrangling with R

Part of the book series: Use R! ((USE R))

  • 13k Accesses

Abstract

Rapid growth of the World Wide Web has significantly changed the way we share, collect, and publish data. Vast amount of information is being stored online, both in structured and unstructured forms. Regarding certain questions or research topics, this has resulted in a new problem—no longer is the concern of data scarcity and inaccessibility but, rather, one of overcoming the tangled masses of online data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    In Automated Data Collection with R Munzert et al. state that “[t]he first way to get data from the web is almost too banal to be considered here and actually not a case of web scraping in the narrower sense.”

  2. 2.

    http://www.bls.gov/cex/pumd_data.htm#csv

  3. 3.

    http://www.bls.gov/data/#prices

  4. 4.

    http://download.bls.gov/pub/time.series/ap/

  5. 5.

    An example is provided in Automated Data Collection with R in which they use a similar approach to extract desired CSV files scattered throughout the Maryland State Board of Elections websiteMaryland State Board of Elections website.

  6. 6.

    You can learn more about selectors at flukeout.github.io

  7. 7.

    You can simply assess the name of the ID in the highlighted element or you can right click the highlighted element in the developer tools window and select Copy selector. You can then paste directly into `html_nodes() as it will paste the exact ID name that you need for that element.

  8. 8.

    See Sect. 16.2.2 Scraping Specific HTML Nodes for details regarding the element selector process.

  9. 9.

    Read more about OAuth athttps://oauth.net/

  10. 10.

    https://cran.r-project.org/web/views/WebTechnologies.html

  11. 11.

    https://ropensci.org/packages/

  12. 12.

    http://stats.stackexchange.com/questions/12670/data-apis-feeds-available-as-packages-in-r

  13. 13.

    http://www.bls.gov/help/hlpforma.htm#ML

  14. 14.

    http://www.ncdc.noaa.gov/cdo-web/webservices/v2

  15. 15.

    https://api.data.gov/docs/ed/

  16. 16.

    https://cran.r-project.org/web/packages/httr/vignettes/quickstart.html

Bibliography

  • Munzert, S., Rubba, C., Meißner, P., & Nyhuis, D. (2014). Automated data collection with R: A practical guide to web scraping and text mining. John Wiley & Sons.

    Google Scholar 

  • Nolan, D., & Lang, D. T. (2014). XML and Web Technologies for Data Sciences with R. Springer.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Air Force Institute of Technology, Dayton, OH, USA

    Bradley C. Boehmke Ph.D.

Authors
  1. Bradley C. Boehmke Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Boehmke, B.C. (2016). Scraping Data. In: Data Wrangling with R. Use R!. Springer, Cham. https://doi.org/10.1007/978-3-319-45599-0_16

Download citation

Publish with us

Policies and ethics

Search

Navigation