# Learn Data Science for Business

## Web Scraping Product Data in R with rvest and purrr

_Written by Joon Im_

This article comes from [Joon Im](https://www.linkedin.com/in/joonhoim/), a student in [Business Science University](https://university.business-science.io/). Joon has completed both the 201 (Advanced Machine Learning with H2O) and 102 (Shiny Web Applications) courses. Joon shows off his progress in this **_Web Scraping Tutorial with `rvest`_**.

### R Packages Covered:

- `rvest` & `jsonlite` - Web Scraping `HTML` and working with `JSON` data
- `purrr` - Iteration through lists using `map()` and `safely()`
- `stringr` - Text manipulation
- `ggplot2` - Data visualization and understanding data

# My Workflow

Here’s a diagram of the workflow I used to web scrape the Specialized Data and create an application:

1. Start with URL of Specialized Bicycles
2. Use `rvest` and `jsonlite` to extract product data
3. Clean up data into “tidy” format using `purrr` and `stringr`
4. Visualize product prices with `ggplot2`
5. Make a `Shiny` Web App using the Business Science 102 Course.

# My Shiny App

I built a `shiny` web application to recommend product prices of new bicycles, which you can try out: [Specialize Product Price Recommendation Application](https://joon.shinyapps.io/specialized_price_prediction/).

I explain more details about how I built my `shiny` app in [Section 5 - Predictive Web App](/content/code-tools/2019-10-07/rvest-web-scraping.html#shiny-app/index.html).

# Tutorial - Web Scraping with rvest

This tutorial showcases how to web scrape websites using `rvest` and `purrr`. I’ll show how to collect data on the 2020 Specialized Bicycles Product Collection, a useful task in **_building a strategic database of product and competitive information for an organization_**.

## 1. Set Up

### 1.1 Introduction

[Specialized®](https://www.specialized.com/us/en/) is a bicycle company founded by Mike Sinyard in 1974 from his hometown of Morgan Hill, California. They became known for creating the first production mountain bike back in 1981, called the _Stumpjumper_. Now they are building professional-grade bikes for riders around the world. Here’s a nice breakdown of different models on [Bike Radar](https://www.bikeradar.com/brand/specialized/) if you are interested in learning more.

One great offering is their ongoing [Learning Labs Pro series](https://university.business-science.io/p/learning-labs-pro), which teaches additional skills such as time series forecasting, customer churn survival analysis, web-scraping and more.

In [Learning Lab 8: Web Scraping — Build A Strategic Database With Product Data](https://university.business-science.io/courses/learning-labs-pro/lectures/9931459) from Business Science, a challenge for students was issued to scrape product data on bikes from Specialized’s website. Today, we’re going to do just that.

### 1.2 Check Robots

Always look at the website’s `robots.txt` to check crawling permissions. Here’s [Specialized’s robots.txt](https://www.specialized.com/us/en/robots.txt).

### 1.3 Load Libraries

Let’s start with loading libraries that we know we will need.

```r
# Load libraries
library(rvest)     # HTML Hacking & Web Scraping
library(jsonlite)  # JSON manipulation
library(tidyverse) # Data Manipulation
library(tidyquant) # ggplot2 theme
library(xopen)     # Opens URL in Browser
library(knitr)     # Pretty HTML Tables
```

### 1.4 Check Out the Products

Let’s navigate to the [“Bikes” Page](https://www.specialized.com/us/en/shop/bikes/c/bikes) for Specialized.

Save the URL.

```r
# URL to View All Bikes
url <- "https://www.specialized.com/us/en/shop/bikes/c/bikes?q=%3Aprice-desc%3Aarchived%3Afalse&show=All"
```

### 1.5 Read HTML

Load the `HTML` code into an object using `read_html()`.

```r
# Read HTML from URL
html <- read_html(url)
h
```

## 2. Get the Raw Data

Use [Chrome DevTools](https://developers.google.com/web/tools/chrome-devtools) to locate the product information. In our case, there is a `JSON`-like dictionary containing what we need.

### 2.1 Locate Data with Chrome DevTools

### 2.2 Find Product Data Nodes

### 2.3 Filter HTML to Isolate Nodes

```r
html %>%
    html_nodes(".product-list__item-wrapper")
```

### 2.4 Extract the Attribute Data

```r
# Store JSON as object
json <- html %>%
    html_nodes(".product-list__item-wrapper") %>%
    html_attr("data-product-ic")

# Show the 1st JSON element (1st bike of 399 bikes)
json[1]
```

```text
{"name":"S-Works Roubaix - SRAM Red eTap AXS","id":"171042","brand":"Specialized","price":11500,"currencyCode":"USD","position":"","variant":"61","dimension1":"Bikes","dimension2":"Road","dimension3":"Roubaix","dimension4":"","dimension5":"Performance Road","dimension6":"S-Works","dimension7":"","dimension8":"Men/Women"}
```

## 3. Format as Tidy Data with purrr

Tidy data is a `tibble` (data frame) that has one row for each of the Specialized Bike Models and columns for each of the features like model name, price, and various categories (denoted as dimensions).

### 3.1 Make a Function that Converts JSON to Tibble

```r
# Make Function
from_json_to_tibble <- function(json) {
    json %>%
        fromJSON() %>%
        as_tibble()
}
```

### 3.4 Extract the Attribute Data

```r
# Store JSON as object
json <- html %>%
    html_nodes(".product-list__item-wrapper") %>%
    html_attr("data-product-ic")

# Show the 1st JSON element (1st bike of 399 bikes)
json[1]
```

### 4. Explore Bike Models

I want to understand how price depends on various features like model, type of bike (electric, mountain, road), and other features that will eventually be used in my `XGBoost` Machine Learning model inside of my `Shiny` Web App.

#### 4.1 Most and Least Expensive Bike Models

```r
bike_features_tbl %>%
    select(dimension3, price) %>%
    mutate(dimension3 = as_factor(dimension3) %>%
               fct_reorder(price, .fun = median)) %>%
    # Plot
    ggplot(aes(dimension3, price)) +
    geom_boxplot() +
    coord_flip() +
    theme_tq()  +
    scale_y_continuous(labels = scales::dollar_format()) +
    labs(title = "Specialized Bike Models by Price")
```

### 5. Predictive Web Application

I made and deployed a [Product Price Recommendation Application for Specialized Bicycles](https://joon.shinyapps.io/specialized_price_prediction/) using the web-scraped Specialized Data. Here’s how I built it:

- The `Shiny` app uses the webscraped data from 2019 Specialized Models (this tutorial covers web-scraping 2020 models), which I learned in [Learning Lab 8](https://university.business-science.io/courses/learning-labs-pro/lectures/9931459).
- The `shiny` application uses an **_`XGBoost` Machine Learning model_** to recommend product prices based on the existing product portfolio.
- The code is available in my [GitHub Repo Here](https://github.com/joon-im/specialized_price_prediction).

# Parting Thoughts

Web-scraping with `rvest` has fundamentally changed the way I understand the Internet. Once I realized that the entire Internet (well, most of it) is basically just one big database, it rocked my world. I highly encourage you to sign up for [Learning Labs Pro](https://university.business-science.io/p/learning-labs-pro). Learning Lab 8 - Web Scraping - Build A Strategic Database With Product Data with `rvest` was what opened my eyes to the power of web scraping.

Using the data, I was able to make and deploy a `Shiny` web application that uses an `XGBoost` Machine Learning model to predict and recommend bicycle prices.