Advertisement

HTTP Requests

  • Deborah Nolan
  • Duncan Temple Lang
Chapter
Part of the Use R! book series (USE R)

Abstract

In this chapter, we focus on general, all-purpose infrastructure we can use in R for accessing networks and the Web. The RCurl package provides both high-level and intermediate-level functionality within R that allow us to make rich and flexible requests to a large variety of different servers, including Web servers and applications that speak different protocols. RCurl generalizes the functionality built into R for downloading documents, etc., by a) supporting more protocols, b) allowing us to control many more aspects of the requests.We use RCurl as the foundation in R for scraping data from static Web pages, submitting HTML forms, interacting with RESTful APIs (Application Programming Interfaces) on both Web servers and local applications, invoking methods provided by SOAP servers, and using XML-RPC for remote procedure calls. RCurl provides support for many protocols used on the Internet (e.g., HTTP, HTTPS, FTP) and a large collection of options for controlling the requests (e.g., cookies, login and passwords, content type, header fields). In this chapter, we will focus primarily on HTTP and illustrate how to use RCurl generally by developing interfaces to different Web APIs.

Keywords

Status Code Internet Protocol Address Secure Socket Layer Multiple Request Character Encode 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    Daniel Adler. rdyncall: Improved foreign function interface (FFI) and dynamic bindings to C libraries (e.g., OpenGL). http://cran.r-project.org/package=rdyncall, 2012. R package version 0.7.5.
  2. [2]
    R. Fielding, J. Gettys, J. Mugul, H. Frystyk, L. Masinter, P. Leach, and T. Berners-Lee. Hypertext Transfer Protocol: HTTP/1.1. Worldwide Web Consortium, 1999. http://www.w3.org/Protocols/rfc2616/rfc2616.html.
  3. [3]
    A. Frank and A. Asuncion. UCI machine learning repository. http://archive.ics.uci.edu/ml, 2010.
  4. [4]
    Robert Gentleman and Ross Ihaka. Lexical scope and statistical computing. Journal of Computational and Graphical Statistics, 9:491–508, 2000.MathSciNetGoogle Scholar
  5. [5]
    GlobalGiving Foundation. Globalgiving: Donate to projects around the world supporting disaster relief, education, health, women and children, and more. http://www.globalgiving.org/, 2012.
  6. [6]
    David Gourley and Brian Totty. HTTP: The Definitive Guide. O’Reilly Media, Inc., Sebastopol, CA, 2002.Google Scholar
  7. [7]
    Michael Nielsen. Reinventing Discovery: The New Era of Networked Science. Princeton University Press, Princeton, NJ, 2012.Google Scholar
  8. [8]
    Open SSL Project. Open SSL: Cryptography and SSL/TLS toolkit. http://openssl.org,2011.
  9. [9]
    Karthik Ram and Duncan Temple Lang. rDrop: Dropbox R interface. https://github.com/karthikram/rDrop/, 2012. R package version 0.3.
  10. [10]
    Leonard Richardson and Sam Ruby. RESTful Web Services. O’Reilly Media, Inc., Sebastopol, CA, 2007.Google Scholar
  11. [11]
    Daniel Steinberg. curl easy setopt — Set options for a curl easy handle. http://curl.haxx.se/libcurl/c/curl_easy_setopt.html, 2012.
  12. [12]
    Daniel Steinberg. libcurl: The multiprotocol file transfer library. http://curl.haxx.se, 2012.
  13. [13]
    Duncan Temple Lang. RGCCTranslationUnit: R interface to GCC source code information. http://www.omegahat.org/RGCCTranslationUnit, 2009. R package version 0.4-
  14. [14]
    Duncan Temple Lang. RGoogleTrends: Download Google Trends data. http://www.omegahat.org/RGoogleTrends, 2009. R package version 0.2-1.
  15. [15]
    Duncan Temple Lang. RCIndex: R interface to the clang parser’s C API. http://www.omegahat.org/RCIndex, 2010. R package version 0.2-0.
  16. [16]
    Duncan Temple Lang. Rffi: Interface to libffi to dynamically invoke arbitrary compiled routines at run-time without compiled bindings. http://www.omegahat.org/Rffi, 2011. R package version 0.3-0.
  17. [17]
    Duncan Temple Lang. XML: Tools for parsing and generating XML within R and S-PLUS. http://www.omegahat.org/RSXML, 2011. R package version 3.4.
  18. [18]
    Duncan Temple Lang. Rcompression: In-memory decompression for GNU zip and bzip2 formats. http://www.omegahat.org/Rcompression, 2012. R package version 0.94-0.
  19. [19]
    Duncan Temple Lang. RCurl: General network (HTTP, FTP, etc.) client interface for R. http://www.omegahat.org/RCurl, 2012. R package version 1.95-3.
  20. [20]
    Duncan Temple Lang. RGoogleDocs: Primitive interface to Google Documents from R. http://www.omegahat.org/RGoogleDocs, 2012. R package version 0.7-0.
  21. [21]
    Duncan Temple Lang. SSOAP: Client-side SOAP access for R. http://www.omegahat.org/SSOAP, 2012. R package version 0.9-0.
  22. [22]
    Twitter, Inc. Twitter: A real-time information network. http://twitter.com/about, 2012.
  23. [23]
    Dave Winer. XML-RPC specification. http://xmlrpc.scripting.com/spec.html, 1999.

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Deborah Nolan
    • 1
  • Duncan Temple Lang
    • 2
  1. 1.Department of StatisticsUniversity of CaliforniaBerkeleyUSA
  2. 2.Department of StatisticsUniversity of CaliforniaDavisUSA

Personalised recommendations