# Learn Data Science for Business

## Become a Data Scientist in our online courses

Earn 6 figures or more in 6 months or less by learning R, Shiny, Machine Learning, Time Series, Web Apps, AWS, Cloud, and more!

5-10 Hours Per Week.  80/20 Skills.  End-To-End Business Projects.

[Start Your Courses Now! 👉](https://university.business-science.io/p/5-course-bundle-machine-learning-web-apps-time-series?el=website)

Join over 100,000+ Data Scientists

# PDF Scraping in R with tabulizer

_Written by Jennifer Cooper_

This article comes from [Jennifer Cooper](https://www.linkedin.com/in/jennifermariecoopermba/), a new student in [Business Science University](https://university.business-science.io/?affcode=173166_fpyudtfo). Jennifer is 35% complete with the 101 course - and shows off her progress in this **_PDF Scraping tutorial_**. Jennifer has an interest in understanding the _plight of wildlife_ across the world, and uses her new data science skills to perform a useful analysis - **_scraping PDF tables of a Report on Endangered Species with the `tabulizer` R package and visualizing alarming trends with `ggplot2`_**.

**R Packages Covered**:

- `tabulizer` - Scraping PDF tables
- `dplyr` - Wrangling unclean data & preparation for data visualization
- `ggplot2` - Data visualization and understanding trends

# My Workflow

Here’s a diagram of the workflow I used:

1. Start with PDF

2. Use `tabulizer` to extract tables

3. Clean up data into “tidy” format using `tidyverse` (mainly `dplyr`)

4. Visualize trends with `ggplot2`

[My Code Workflow for PDF Scraping with `tabulizer`](/content/code-tools/2019/09/23/tabulizer-pdf-scraping.html#workflow/index.html)

# Get the PDF

I analyzed the [Critically Endangered Species PDF Report](https://github.com/Coopmeister/data_science_r_projects/blob/master/endangered_species.pdf).

# PDF Scrape and Exploratory Analysis

## Step 1 - Load Libraries

Load the following libraries to follow along.

```r
library(rJava)      # Needed for tabulizer
library(tabulizer)  # Handy tool for PDF Scraping
library(tidyverse)  # Core data manipulation and visualization libraries
```

Note that `tabulizer` depends on `rJava`, which may require some setup. Here are a few pointers:

- **Mac Users:** If you have issues connecting `Java` to `R`, you can try running `sudo R CMD javareconf` in the Terminal ( [per this post](https://github.com/rstudio/rstudio/issues/2254))
- **Windows Users:** [This blog article](https://cimentadaj.github.io/blog/2018-05-25-installing-rjava-on-windows-10/installing-rjava-on-windows-10/) provides a step-by-step process for installing `rJava` on Windows machines.

## Step 2 - Extracting the Tabular Data from PDF

The `tabulizer` package provides a suite of tools for extracting data from PDFs. The vignette, [“Introduction to tabulizer”](https://cran.r-project.org/web/packages/tabulizer/vignettes/tabulizer.html) has a great overview of `tabulizer`’s features.

We’ll use the `extract_tables()` function to pull out each of the tables from the Endangered Species Report. This returns a `list` of `data.frames`.

```r
# PDF Scrape Tables
dangered_species_scrape <- extract_tables(
    file   = "2019-09-23-tabulizer/endangered_species.pdf",
    method = "decide",
    output = "data.frame")
```

## Step 3 - Clean Up Column Names

Next, I want to start by cleaning up the names in my data - which are actually in the first row. I’ll use a trick using `slice()` to grab the first row, and the new `pivot_longer()` function to transpose and extract the column names that are in row 1. I can then `set_names()` and remove row 1.

```r
# Get column names from Row 1
col_names <- endangered_species_raw_tbl %>%
    slice(1) %>%
    pivot_longer(cols = everything()) %>%
    mutate(value = ifelse(is.na(value), "Missing", value)) %>%
    pull(value)

# Overwrite names and remove Row 1
endangered_species_renamed_tbl <- endangered_species_raw_tbl %>%
    set_names(col_names) %>%
    slice(-1)
```

## Step 4 - Tidy the Data

There are a few issues with the data:

1. **Remove columns with NAs:** Column labelled “Missing” is all NA’s - We can just drop this column
2. **Fix columns that were combined**: Three of the columns are combined - Amphibians, Fishes, and Insects - We can `separate()` these into 3 columns
3. **Convert to (Tidy) Long Format for visualization**: The data is in “wide” format, which isn’t tidy - We can use `pivot_longer()` to convert to “long” format with one observation for each row
4. **Fix numeric data stored as character**: The numeric data is stored as character and several of the numbers have commas - We’ll remove commas and convert to numeric
5. **Convert Character Year & species to Factor**: The year and species columns are character - We can convert to factor for easier adjusting of the order in the ggplot2 visualizations
6. **Percents by year**: The visualizations will have a percent (proportion) included so we can see which species have the most endangered - We can add proportions by each year

```r
endangered_species_final_tbl <- endangered_species_renamed_tbl %>%
    # 1. Remove columns with NAs
    select_if(~ !all(is.na(.))) %>%
    # 2. Fix columns that were combined
    separate(col  = `Amphibians Fishes Insects`,
             into = c("Amphibians", "Fishes", "Insects"),
             sep  = " ") %>%
    # 3. Convert to (Tidy) Long Format for visualization
    pivot_longer(cols = -Year, names_to = "species", values_to = "number") %>%
    # 4. Fix numeric data stored as character
    mutate(number = str_remove_all(number, ",")) %>%
    mutate(number = as.numeric(number)) %>%
    # 5. Convert Character Year & species to Factor
    mutate(Year = as_factor(Year)) %>%
    mutate(species = as.factor(species)) %>%
    # 6. Percents by year
    group_by(Year) %>%
    mutate(percent = number / sum(number)) %>%
    mutate(label = scales::percent(percent)) %>%
    ungroup()
```

## Step 5 - Visualize the Data

### Summary Visualization

I made a summary visualization using stacked bar chart to show the alarming trends of critically endangered species over time.

```r
endangered_species_final_tbl %>%
    mutate(Year = fct_rev(Year)) %>%
    ggplot(aes(x = Year, y = number, fill = species)) +
    # Geoms
    geom_bar(position = position_stack(), stat = "identity", width = .7) +
    geom_text(aes(label = label), position = position_stack(vjust= 0.5), size = 2) +
    coord_flip() +
    # Theme
    labs(
        title = "Critically Endangered Species",
        y = "Number of Species Added to Critically Endangered List", x = "Year"
    ) +
    theme_minimal()
```

### Trends Over Time by Species

I then faceted the species and visualized the trend over time using a smoother (`geom_smooth`). Again, we see that each of the species exhibit increasing trends.

```r
endangered_species_final_tbl %>%
    mutate(Year = fct_rev(Year)) %>%
    # Geom
    ggplot(aes(Year, number, color = species, group = species)) +
    geom_point() +
    geom_smooth(method = "loess") +
    facet_wrap(~ species, scales = "free_y", ncol = 3) +
    # Theme
    expand_limits(y = 0) +
    theme_minimal() +
    theme(legend.position = "none",
          axis.text.x = element_text(angle = 45, hjust = 1)) +
    labs(
        title = "Critically Endangered Species",
        subtitle = "Trends Not Improving",
        x = "", y = "Changes in Number of Species in Threatened Category"
    )
```

# Parting Thoughts

**It was really exciting to see my hard work pay off.** It took a bit to get going, but I found that `tabulizer` made PDF extraction manageable. The most challenging part was getting the data into a format that can be easily visualized (the `tidyverse` really helped as shown in Step 4!). I was particularly excited to see results of my analysis. I want to share with others the alarming trends related to the plight of wildlife, while demonstrating the power of `R`!

**If you’d like to join me**, I’m currently learning Data Science for Business in [Business Science’s 101 course (Data Science Foundations)](https://university.business-science.io/p/ds4b-101-r-business-analysis-r/?coupon_code=ds4b15&affcode=173166_fpyudtfo).

# My Workflow

Here’s a diagram of the workflow I used:

1. Start with PDF

2. Use `tabulizer` to extract tables

3. Clean up data into “tidy” format using `tidyverse` (mainly `dplyr`)

4. Visualize trends with `ggplot2`