Learn Data Science for Business

Web Scraping Product Data in R with rvest and purrr

Written by Joon Im

This article comes from Joon Im, a student in Business Science University. Joon has completed both the 201 (Advanced Machine Learning with H2O) and 102 (Shiny Web Applications) courses. Joon shows off his progress in this Web Scraping Tutorial with rvest.

R Packages Covered:

  • rvest & jsonlite - Web Scraping HTML and working with JSON data
  • purrr - Iteration through lists using map() and safely()
  • stringr - Text manipulation
  • ggplot2 - Data visualization and understanding data

My Workflow

Here’s a diagram of the workflow I used to web scrape the Specialized Data and create an application:

  1. Start with URL of Specialized Bicycles
  2. Use rvest and jsonlite to extract product data
  3. Clean up data into “tidy” format using purrr and stringr
  4. Visualize product prices with ggplot2
  5. Make a Shiny Web App using the Business Science 102 Course.

My Shiny App

I built a shiny web application to recommend product prices of new bicycles, which you can try out: Specialize Product Price Recommendation Application.

I explain more details about how I built my shiny app in Section 5 - Predictive Web App.

Tutorial - Web Scraping with rvest

This tutorial showcases how to web scrape websites using rvest and purrr. I’ll show how to collect data on the 2020 Specialized Bicycles Product Collection, a useful task in building a strategic database of product and competitive information for an organization.

1. Set Up

1.1 Introduction

Specialized® is a bicycle company founded by Mike Sinyard in 1974 from his hometown of Morgan Hill, California. They became known for creating the first production mountain bike back in 1981, called the Stumpjumper. Now they are building professional-grade bikes for riders around the world. Here’s a nice breakdown of different models on Bike Radar if you are interested in learning more.

One great offering is their ongoing Learning Labs Pro series, which teaches additional skills such as time series forecasting, customer churn survival analysis, web-scraping and more.

In Learning Lab 8: Web Scraping — Build A Strategic Database With Product Data from Business Science, a challenge for students was issued to scrape product data on bikes from Specialized’s website. Today, we’re going to do just that.

1.2 Check Robots

Always look at the website’s robots.txt to check crawling permissions. Here’s Specialized’s robots.txt.

1.3 Load Libraries

Let’s start with loading libraries that we know we will need.

# Load libraries
library(rvest)     # HTML Hacking & Web Scraping
library(jsonlite)  # JSON manipulation
library(tidyverse) # Data Manipulation
library(tidyquant) # ggplot2 theme
library(xopen)     # Opens URL in Browser
library(knitr)     # Pretty HTML Tables

1.4 Check Out the Products

Let’s navigate to the “Bikes” Page for Specialized.

Save the URL.

# URL to View All Bikes
url <- "https://www.specialized.com/us/en/shop/bikes/c/bikes?q=%3Aprice-desc%3Aarchived%3Afalse&show=All"

1.5 Read HTML

Load the HTML code into an object using read_html().

# Read HTML from URL
html <- read_html(url)
h

2. Get the Raw Data

Use Chrome DevTools to locate the product information. In our case, there is a JSON-like dictionary containing what we need.

2.1 Locate Data with Chrome DevTools

2.2 Find Product Data Nodes

2.3 Filter HTML to Isolate Nodes

html %>%
    html_nodes(".product-list__item-wrapper")

2.4 Extract the Attribute Data

# Store JSON as object
json <- html %>%
    html_nodes(".product-list__item-wrapper") %>%
    html_attr("data-product-ic")

# Show the 1st JSON element (1st bike of 399 bikes)
json[1]

{"name":"S-Works Roubaix - SRAM Red eTap AXS","id":"171042","brand":"Specialized","price":11500,"currencyCode":"USD","position":"","variant":"61","dimension1":"Bikes","dimension2":"Road","dimension3":"Roubaix","dimension4":"","dimension5":"Performance Road","dimension6":"S-Works","dimension7":"","dimension8":"Men/Women"}

3. Format as Tidy Data with purrr

Tidy data is a tibble (data frame) that has one row for each of the Specialized Bike Models and columns for each of the features like model name, price, and various categories (denoted as dimensions).

3.1 Make a Function that Converts JSON to Tibble

# Make Function
from_json_to_tibble <- function(json) {
    json %>%
        fromJSON() %>%
        as_tibble()
}

3.4 Extract the Attribute Data

# Store JSON as object
json <- html %>%
    html_nodes(".product-list__item-wrapper") %>%
    html_attr("data-product-ic")

# Show the 1st JSON element (1st bike of 399 bikes)
json[1]

4. Explore Bike Models

I want to understand how price depends on various features like model, type of bike (electric, mountain, road), and other features that will eventually be used in my XGBoost Machine Learning model inside of my Shiny Web App.

4.1 Most and Least Expensive Bike Models

bike_features_tbl %>%
    select(dimension3, price) %>%
    mutate(dimension3 = as_factor(dimension3) %>%
               fct_reorder(price, .fun = median)) %>%
    # Plot
    ggplot(aes(dimension3, price)) +
    geom_boxplot() +
    coord_flip() +
    theme_tq()  +
    scale_y_continuous(labels = scales::dollar_format()) +
    labs(title = "Specialized Bike Models by Price")

5. Predictive Web Application

I made and deployed a Product Price Recommendation Application for Specialized Bicycles using the web-scraped Specialized Data. Here’s how I built it:

  • The Shiny app uses the webscraped data from 2019 Specialized Models (this tutorial covers web-scraping 2020 models), which I learned in Learning Lab 8.
  • The shiny application uses an XGBoost Machine Learning model to recommend product prices based on the existing product portfolio.
  • The code is available in my GitHub Repo Here.

Parting Thoughts

Web-scraping with rvest has fundamentally changed the way I understand the Internet. Once I realized that the entire Internet (well, most of it) is basically just one big database, it rocked my world. I highly encourage you to sign up for Learning Labs Pro. Learning Lab 8 - Web Scraping - Build A Strategic Database With Product Data with rvest was what opened my eyes to the power of web scraping.

Using the data, I was able to make and deploy a Shiny web application that uses an XGBoost Machine Learning model to predict and recommend bicycle prices.