Advertisement

Practical R 4 pp 293-311 | Cite as

Project 6: The R Personal Assistant

  • Jon Westfall
Chapter
  • 63 Downloads

Abstract

If you are like me, you probably have a morning routine that involves information. Perhaps before you even get out of bed, you grab your phone or tablet and hit your “cycle” of apps: email, news, social media, and perhaps company reports and analytics. Sometimes it seems like you spend more of your time cycling through things than actually reading them. Wouldn’t it be nice to tie all of that together? In this chapter, we’ll do exactly that by having R build a daily briefing for us, and we’ll discuss ways to get it to us in the morning (or anytime we need it) as quickly and easily!

If you are like me, you probably have a morning routine that involves information. Perhaps before you even get out of bed, you grab your phone or tablet and hit your “cycle” of apps: email, news, social media, and perhaps company reports and analytics. Sometimes it seems like you spend more of your time cycling through things than actually reading them. Wouldn’t it be nice to tie all of that together? In this chapter, we’ll do exactly that by having R build a daily briefing for us, and we’ll discuss ways to get it to us in the morning (or anytime we need it) as quickly and easily!

Planning Part 1: Choosing Your Sources

Each person reading this chapter will probably have a different idea of what is essential and vital to them every morning. When planning the chapter, the following thoughts came to my mind:
  • I generally need to know what the weather is for the day.

  • I always check my calendar to see how long I can stay in bed before my first meeting!

  • I tend to be curious about how some of my investments are doing – the most recent stocks I’ve bought, or the market in general.

  • Reviewing the top news stories is always important, especially when you don’t watch or listen to news.

  • It would be nice to start things off with a bit of inspiration as well.

To a lesser degree, it might be important to someone to
  • Receive reports from their analytics software, perhaps most recently software or unit sales

  • Have a summary of social media interactions

  • Have their to-do list items present and accounted for

  • Hear about specific news items such as content about their local town or favorite sports team

  • Monitor the progress of ongoing jobs on their computers, such as backups or network activity

And the list could go on and on. For the following example, I’ve focused on the five items that were in my first list. But the second list is completely doable as well; it just might take some ingenuity. Let’s take a moment to think about where we get the data that goes into these lists:
  1. 1.

    Through API calls, like we saw with the Reddit examples in Chapters  8 and  9.

     
  2. 2.

    Through web scraping, for when an API is not available.

     
  3. 3.

    Through internal reports – you may have a piece of software that runs nightly and creates a report, dropped onto your hard drive or RStudio Server.

     
  4. 4.

    Through command-line calls – you can use something like the R ssh package to run a command on a remote computer and capture the results, perhaps the uptime command on all of your machines to verify their load levels and status.

     
  5. 5.

    Through communication from others – you could draw in emails and tweets from your friends to put in your report as well.

     

Each of those items is going to require a bit of planning to pull off. For our example, we’ve limited ourselves to the API calls; however, you can imagine using the code examples from previous chapters in this book to scrape websites and create reports. If you wanted to be especially daring, you could script an email to be sent out every afternoon at 4:30 PM to your employees reminding them to fill out a form with the day’s report. You could then have an R script download that form data (see Chapters  2,  3, and  7) every evening and compile a report. You could finally have your daily briefing script at 8:00 AM every morning pull portions of that report to put into your briefing. All of this could happen after you programmed it one time, without you having to chase down reports every day or week and compile them by hand.

Once you’ve decided what sources you want to use, your next step is to figure out the format you’d like your daily report to be in. Let’s talk about that next.

Planning Part 2: Choosing Your Format

As we have seen, R can output information into many different formats. For your daily report, you might want to consider the following options. I’ve discussed each one in the following and noted the pros and cons of it: plain text, HTML/PDF, or data frame/graphs.

Plain Text

Perhaps the simplest and most effective way to communicate in the world is just through words. It’s how I’m communicating in this book, and it’s how you’re taking in the information. While infographics, charts, spreadsheets, and PowerPoint slides are useful, sometimes they add more clutter than information. Plain old text can sometimes be king for its brevity, its ability to be specific using small changes of phrase, its reliability, and small size. Plain text is also extremely easy for a text-to-speech (TTS) synthesizer to use. Thus, your smartphone or personal assistant can easily read your daily report without having issues navigating around photos, images, and formatting blocks.

However, text can fall short in some key spaces. First, for very complex thoughts, a picture can really be worth a thousand words. Thinking about how I explain certain statistics topics, such as three-way interactions, the graphs I use are way more intuitive than simply explaining it in words. Plus, there are certain things I cannot communicate in text. If I’m a product designer and part of my job is to review the latest design sketches every morning, I really can’t depend on text for my daily briefing. And finally, text does take longer to parse than many other formats. Indeed you may notice in the daily briefing script as follows that while the text version of the script is shorter in length, the visual appeal of the HTML/PDF version makes it more palatable to the eye.

HTML/PDF

Rendering text in a rich environment while also adding formatting, images, and color can produce a stylish and professional looking report that you will enjoy reading each morning. It also has the added benefit that it’s a report you will probably not hesitate to share if needed – especially if it’s information that could appeal to others in your circle of friends or co-workers. UI/UX designers use the term skeuomorphic to describe the emulation of a real-world object virtually, such as the older version of applications on iOS that looked like their paper counterparts (e.g., notes, contacts, and calendar).

While not many of us will go “all out” with our skeuomorphic designs, there is something to be said about beauty as well as information. One example of this is the Tufte Handout style that one can actually create in RStudio using RMarkdown. This style is beautiful and simple and harkens back to a clean textbook style of decades past. However, while it may be visually appealing, a TTS synthesizer may have issues with it, and it could also be overwhelming to the reader who just wants to get what they need and get done with the document. Pretty may equal professional, but it doesn’t always equal fast.

Data Frame/Graphs

Our last manner of rendering data is probably the most well developed in R, given its roots as a statistical software package, but also the trickiest to pull off effectively: exporting data as a data frame or in graphical representation. I won’t spend a lot of time on this here, because it’s likely that if your daily briefing depends on statistics and numbers in columns, you probably already know exactly what graphs you want and what tables of text you expect to see. I will note that some of the formatting that we’ll see in the HTML version of the daily briefing R script is very useful if you are going to be working with these areas. And don’t forget the officer package we talked about in Chapter  7 – it can be used to write and read Excel workbooks. I shared an example of an office having to put together the same report daily for 2 weeks and how this could be easily done using officer. I could see a similar scenario for someone in an organization that reviews metrics each morning designing an R script to create an Excel workbook with several sheets that could be shared.

Once you have decided what format you want your daily report in, it’s time to glue it all together. In the next section, I’ll walk through the daily briefing R script, how it pulls in its data, and how it exports data out in either plain text or in HTML through RMarkdown. Once we’re doing walking through it, I’ll talk about how we might distribute it to ourselves for the easiest “digestion” of our briefing.

Bringing It Together: The Daily Briefing R Script

At its core, the daily briefing R script ties together a lot of what we’ve seen already in this book; however, it does bring together more elements at one time than we’ve seen before. Thankfully, they all work together fairly seamlessly. I’ll discuss each of the sources first, and then we’ll build the plain text output and the RMarkdown output.

Packages You’ll Need

Make sure you’ve installed the following R packages and their dependents before running the following code:
  • owmr

  • tidyquant

  • jsonlite

  • devtools (to install the Google Calendar library, gcalendr)

  • markdown

  • knitr

Weather from OpenWeatherMap.Org

Weather is a necessary consideration for most of us when we’re not sheltering in place or quarantined (as you can imagine, during the month I’m writing this, April 2020, I’m not as interested in the weather as I usually am). OpenWeatherMap.org offers several weather APIs that you can choose from, with their basic forecast API for free for up to 1000 calls per day. You will need to sign up and get an API key in order to access their data remotely. Once you have yours, you’ll need to replace the example APPID that I have in the following code. You’ll also want to change your ZIP code to your location. Their API also allows you to specify city and latitude/longitude.
library('owmr')
Sys.setenv(OWM_API_KEY = "a91bb498c90bcd09f445039a88474ec5")
weather <- get_current(zip="38733", units="imperial")
weather$main$temp
Figure 10-1

Downloaded Weather JSON Object

As you can see from the last line of code, we now have a data frame named weather that contains our forecast data. Later we’ll mine out the current weather conditions to put in our report. And while we’ll only be using two of the variables that we’re given, we could use much more. See Figure 10-1 to see all of the information that was downloaded through OpenWeatherMap.org’s API.

Stock Quotes from quantmod/tidyquant Packages

Grabbing our financial data comes via the well-established tidyquant package that will install quantmod along with it. These pieces of financial data come from Yahoo! Finance. Here we are grabbing the current stock quote; however, we could expand out to get as many ticker symbols as we like, and as you’ll find investigating the quantmod package, you can also get options, dividends, financials, splits, and more data as you find it useful.

library("tidyquant")
dji <- getQuote("^DJI") #Get Current Dow Jones Industrial Average
aapl <- getQuote("AAPL")
goog <- getQuote("GOOG")

Previously, we’ve loaded the library and downloaded three quotes – the current Dow Jones Industrial Average, Apple, and Alphabet Inc. (Google). We’ll slot them into our report as we find it useful. Figure 10-2 shows an example of the data downloaded for one quote, in this case, for Apple.
Figure 10-2

Apple Stock Quote

News Headlines from NewsAPI.org

The world we live in is changing at light speed, so knowing what’s going on can be crucial. NewsAPI.org provides a free developer account that allows you to get live headlines and articles with a 15-minute delay up to 500 requests per day. Definitely, a really nice deal when you realize you can also search their API for news that fits a certain keyword, publishing date, source (by name or domain name), and language. You’ll need to sign up to get an API key from their website and replace my “APIKEY” example here with yours. Once you do that, the following code will download the top 20 headlines in the United States, saving it to a data frame (see Figure 10-3):
library("jsonlite")
url <- "http://newsapi.org/v2/top-headlines?country=us&apiKey=APIKEY"
news.df <- fromJSON(url);
Figure 10-3

Headlines from NewsAPI.org

Daily Inspiration Quote from TheySaidSo.com

After reading the news, you may be feeling a little depressed. TheySaidSo.com has an API that allows you to pull ten calls per hour to get their Quote of the Day, which is all we really need. And it doesn’t even require an API key. The following code will pull it and store it in a data frame for us (see Figure 10-4):
library("jsonlite")
url <- "http://quotes.rest/qod.json?category=inspire"
quote.df <- fromJSON(url);
Figure 10-4

The Daily Quote Downloaded

Calendar Information from Google Calendar

The last piece of the puzzle can also be the most useful – our calendar data from Google Calendar. To get up and running with this code, you’ll need to enable the Google Calendar API (see Figure 10-5) and download an OAuth 2.0 Client Key. Thankfully, the steps for that are almost identical to the steps we took in Chapter  6 to set up gmailr email sending – so just follow those and enable the Google Calendar API instead of the Gmail API. Once you have your client_secret.json file, you can put it into your working directory and run this code. The first time that you authenticate, you will have to allow the unverified application, same as you did in Chapter  6. From that point on, you’re able to download your calendar events.
Figure 10-5

The Google Calendar API

If you have multiple calendars, use the calendar_list() function to get a list of them, and then replace the my_cal_id with the ID of the calendar you’d like to pull. Figure 10-6 shows the day I used for my example throughout this chapter (ironically filled with things I would do on an average Sunday, but didn’t do on this particular Sunday!).

Finally, the last line of code here sorts the events in order from earliest to latest; otherwise, they’d be alphabetical, which might cause your After Dinner Conference Call to show up first on your list before your Zither Breakfast Meditation!
devtools::install_github("andrie/gcalendr")
timezoneoffset <- -4 #Set to your distance from UTC
library(gcalendr)
calendar_auth(email="examples@gmail.com",path="client_secret.json")
my_cal_id <- "examples@gmail.com"
events <- calendar_events(my_cal_id, days_in_past = 0, days_in_future = 1)
events <- events[order(events[,3]),]
Figure 10-6

The Day I used, As Viewed in Google Calendar

With all of the piece put together, we can now create our text report! And we’ll do that in a very basic yet easily changeable way – by building a paragraph.

Plain Text Output

The following code may look rather intimidating, but at its core, it is simply building a paragraph, line by line. Each paste(txt line uses the output from the previous line (txt as its starting point. In some cases, we use sep="" to override paste()s default behavior of putting in an extra space between each element. In other cases, we want the space, so we leave the sep option out.

Reading through this, you’ll notice a few features. First, you can re-arrange the order with ease by cutting and pasting the various blocks. If you don’t want the Calendar Block first, you can put it after the stock or world news block. If you want the URLs for the news articles, you could create a new block at the end and paste them out. And you could always add and remove content as you need to.

Finally, you’ll notice a few nice touches – first, the Calendar Block will format the text properly depending on if this is one of many items or the last item in the list (e.g., by adding the “, and” and by placing a period at the end instead of a comma). The Stock Block also checks to see if the percentage change was positive or negative, so it knows whether to say increased or decreased.

txt <- paste("This is your automated daily briefing for ",format(Sys.time(),'%a %b %d %Y'),"! Today the high temperature will be",sep="")
txt <- paste(txt,as.character(round(weather$main$temp,digits=0)))
txt <- paste(txt," degrees and ",weather$weather$main,".",sep="")
# Calendar Block
txt <- paste(txt,sprintf("You have %s events today.",nrow(events)))
txt <- paste(txt,"The first event,", events$summary[1], ",starts at",format(events$start_datetime[1]+timezoneoffset*60*60,'%H:%M'))
i <- nrow(events)-1
txt <- paste(txt,sprintf(".The remaining %s events are: ",sprintf(as.character(i))))
j <- 2
while (j <= nrow(events)) {
  txt <<- (paste(txt,ifelse(j == nrow(events)," and ",""),events$summary[j]," at ",format(events$start_datetime[j]+timezoneoffset*60*60,'%H:%M'), ifelse(j == nrow(events),".",","),sep=""));
  j <<- j+1
}
# Stock Block
txt <- paste(txt," In stock news, the Dow Jones closed at ",as.character(dji$Last)," a change of ",as.character(round(dji$`% Change`,digits=2))," percent.",
if (aapl$Change > 0) { txt <<- paste(txt,"Apple Stock increased") } else { txt <<- paste(txt,"Apple stock decreased") }
txt <- paste(txt,as.character(round(abs(aapl$`% Change`),digits=2)), " percent. Google stock ")
if (goog$Change > 0) { txt <<- paste(txt,"increased") } else { txt <<- paste(txt,"decreased") }
txt <- paste(txt,as.character(round(abs(goog$`% Change`),digits=2)), " percent.")
# World News Block
txt <- paste(txt,"In world news, the top 3 headlines and their sources are:",news.df$articles$title[1],"followed by",news.df$articles$title[2],"and finally",
# Optionally, include news.df$articles$url at the bottom of your text briefing so you can easily get to the full story.
# Inspirational Quote Block
txt <- paste(txt,". Finally, to wrap up your briefing, today's inspirational quote comes from ",quote.df$contents$quotes$author[1], "and is '", quote.df$

When I ran this code today, my output looked like this:

This is your automated daily briefing for Sun Apr 19 2020! Today the high temperature will be 63 degrees and Clouds. You have 3 events today. The first event, Lunch with Karey, starts at 12:00 .The remaining 2 events are: Podcasting (Mobileviews.com) with Todd at 14:00, and FaceTime with Nate & Kristen at 19:30. In stock news, the Dow Jones closed at 24242.49 a change of 2.99 percent. Apple stock decreased 1.36 percent. Google stock increased 1.57 percent. In world news, the top 3 headlines and their sources are: As coronavirus cases rise in U.S. hot spots, governors tell Trump it's too soon to reopen America - Reuters followed by The coronavirus pandemic will likely leave a lasting legacy on retail: Fewer department stores - CNBC and finally Pence defends Trump's 'LIBERATE' tweets - POLITICO . Finally, to wrap up your briefing, today's inspirational quote comes from Brene Brown and is ' What's the greater risk? Letting go of what people think or letting go of how I feel, what I believe, and who I am? '. That's today's briefing, have a great day, Jon

For good measure, I’ll use the following code to save that output to a text file, which I can easily distribute later. I’ll also save my workspace image to a file, so that I can save some time (and API calls) later.

fileDB <-file("db.txt")
writeLines(txt, fileDB)
close(fileDB)
save.image("daily-brief")

While text is perhaps not the prettiest, it is easy for a text-to-speech synthesizer to read to me. If I want it prettier, well, then I’ll just create an HTML version!

The HTML Markdown Version

To create the more polished version, we’ll use RMarkdown, which we briefly discussed at the end of Chapter  5. Markdown, with R integrated into it, can make for some extremely rapid report writing, as you can see in the following code:

---
title: "Daily Briefing"
output: html_document
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
load("daily-brief")
```
## `r format(Sys.time(),'%a %b %d %Y')`
Welcome to your daily briefing. Today the high temperature will be `r round(weather$main$temp,digits=0)` degrees and `r weather$weather$main`.
```{r echo=FALSE, results="asis"}
library(knitr)
kable(events[c(1,2,3,5)], caption="Your Schedule")
```
In world news, the top 5 headlines and their sources are:
```{r echo=FALSE, results="asis"}
cat(paste(
  '<ul><li><a href=',
  news.df$articles$url[1],'>',
  news.df$articles$title[1],'</a></li><li><a href=',
  news.df$articles$url[2],'>',
  news.df$articles$title[2],'</a></li><li><a href=',
    news.df$articles$url[3],'>',
  news.df$articles$title[3],'</a></li><li><a href=',
    news.df$articles$url[4],'>',
  news.df$articles$title[4],'</a></li><li><a href=',
  news.df$articles$url[5],'>',
  news.df$articles$title[5],'</a></li></ul>'))
```
Here are your stock quotes from the last trading day:
```{r echo=FALSE, results="asis"}
stocks <- rbind(dji,aapl,goog)
kable(stocks)
```
Finally, to wrap up your briefing, today's inspirational quote:
>`r quote.df$contents$quotes$quote` -*`r quote.df$contents$quotes$author`*

Walking through the code, we can see that it starts with a commented title and output type. RMarkdown then sets options by using the triple back-tick (`) to indicate code and then putting a brace ({) and the letter r – this tells the R compiler that it shouldn’t just show this code, it should actually execute it. We can see an example of r code execution in Line 11, where we put in the date of the report. We can also see it on Line 13 when we slip in the temperature and the weather conditions.

Moving into Lines 15–18, we can see a nicely formatted table (using the kable() function). In Lines 20–35, we iterate out the five top stories. We could do this with a loop as well; however, I wrote it out in longer form for easy modification. But it would be more efficient to do this with a for() loop as we did with the text output earlier.

Finally, we finish with our stock quotes and our inspirational quote in block quote format. With the file completely written, all that’s left to do is write the Markdown and HTML versions of my report, using the following code:
## Write the Markdown Version
knitr::knit(input="daily-briefing.Rmd")
## Write the HTML version
library(markdown)
markdownToHTML("daily-briefing.md",output = "daily-briefing.html")
Figure 10-7

The HTML Version

Opening up the HTML version (Figure 10-7) in a web browser, we can see that it’s nicer than the text (and more functional, since I can click the news headlines) – not as easy to read if you’re a computerized voice in a smart speaker, but easier on human eyes. With that, we have two versions of our daily briefing. I could easily set up an R script to run db.R every morning at a set time and have the two files written on my RStudio Server or my desktop and ready to go whenever I wished. Now the final question is – where? Where do I put them so that I can access them when I need them?

Distribution Options

Once you have your daily briefing, it’s up to you how you would like to either access it or have it delivered to yourself. Depending on your workflow, you may be more comfortable with a “pull” method (where you go out and initiate the action of reading the report) or a “push” method (where it’s sent to you automatically). Here are a few options for both that you might want to consider.

Pull Methods

  • Perhaps the most basic would be to have it write the file to a spot on your hard drive that you would access each morning. It may be easiest to have this in a cloud storage folder, such as OneDrive, Dropbox, or Google Drive. That way you could also easily access it from your phone or tablet.

  • Saving it to a dedicated “notes” application such as OneNote or Evernote may also be an option. Some of these apps support “watch folders” – folders that they monitor in the background and look for changes within. If your software supports that, you could easily have the R script write it to that location, and let the note software do the rest.

  • Saving it to a public location, such as your web server. This is likely easy but not very secure unless you secure with a username and password. However, if your daily briefing contains no sensitive information (e.g., perhaps you take out the schedule), then there would be no harm in putting it somewhere that someone could stumble upon it. It might even allow you to create a record of them for future posterity.

  • Speaking of posterity, if you use any sort of journaling or diary application, you might find it useful to have R write directly to it. For example, the application that I use for my personal journal, Day One, has a dedicated command-line interface. I could use that plus the system() function in R to auto import my daily briefing to my journal – a great way to both access it and archive it for the future.

  • To make pulling your report easier, you will likely want to create a shortcut in some way depending on the device you’re using. On iOS, for example, you might use Siri Shortcuts to create a custom home icon that will take you to your report location. In Windows, you might create a shortcut to the file.

Push Methods

  • You could attach your daily report to an email as seen in Chapter  6 and have it emailed to you daily. This also will be useful for archival purposes as well.

  • You could use Pushover as discussed in Chapter  9 to alert you to your daily briefing. Pushover supports image attachments, HTML, and messages up to 1024 characters. My text version earlier was 1070 characters, which would be just a little too long. However, if you shorten your briefing, you may be able to put the whole thing in one Pushover message. At the least, you could include a URL to wherever you publish your briefing.

  • Various text-to-speech options exist. For example, a quick Siri Shortcut on iOS can be created to download and read your report to you (Figure 10-8). Scheduling these can be tricky though. Initially, I wanted to include code in this chapter to have your Amazon Echo device automatically read your briefing to you each morning. This proved to be nearly impossible for an independent, small, home-brew project. The only API, at this time of writing, that would allow this would be the Echo Announce feature, which (because it interrupts you and reads an announcement) is restricted to verified and validated apps. As a completely automated workaround, you could have a text-to-speech engine convert the text to a Wav or MP3 file and use it as your alarm clock tone.

  • Depending on your financial resources and position in society, you might also be able to get this information pushed to you via an administrative assistant or secretary. They could receive the report emailed to them every morning and have it printed or loaded up for you when you get into the office.

Figure 10-8

The Siri Shortcut to Read Your Report to You

Conclusion

As you can see, having your own daily briefing facilitated by R can be a fun project to build out and create as well as a useful timesaver for you during the day. As the final project in this book, I hope that it’s inspired you beyond the limitations of what I’ve shown.

This book has had a central goal of introducing you to ways to use R in your life that you might not have considered. I would challenge you to dream up ways to use the building blocks that we’ve seen, whether they be statistical analysis and basic data manipulation, to offloading daily work to R automation solutions, to allow you to spend more time creating and enjoying the fruits of your creation, rather than simply being a slave to pointing, clicking, and typing the same things over and over again. If that’s been accomplished, even if by just inspiring you, then I’ve done my job. I look forward to hearing from you through your feedback to my social media accounts (principally @jonwestfall on Twitter, a full list available at http://jonwestfall.com/contact) and old-fashioned communication such as email. And if you’re looking for new ideas, you can always check out my blog at http://jonwestfall.com and hear my thoughts nearly every week on the MobileViews podcast (mobileviews.com) with my good friend, Todd Ogasawara. Keep learning, keep creating, and keep growing!

Copyright information

© Jon Westfall 2020

Authors and Affiliations

  • Jon Westfall
    • 1
  1. 1.Division of Counselor Education & Psychology, Division of Student Success CenterDelta State UniversityClevelandUSA

Personalised recommendations