Learn Data Science for Business

Become a Data Scientist in our online courses

Earn 6 figures or more in 6 months or less by learning R, Shiny, Machine Learning, Time Series, Web Apps, AWS, Cloud, and more!

5-10 Hours Per Week.  80/20 Skills.  End-To-End Business Projects.

Start Your Courses Now! 👉

Join over 100,000+ Data Scientists

How to Scrape PDF Documents and Search PDFs with OpenAI LLMs (in R)

Written by Matt Dancho

Hey guys, welcome back to my R-tips newsletter. Businesses are sitting on a mountain of unstructured data. The biggest culprit is PDF Documents. Today, I’m going to share how to PDF Scrape text and use OpenAI’s Large Language Models (LLMs) to summarize it in R.

Table of Contents

Here’s what you’re learning today:

  • How to scrape PDF Documents I’ll explain how to scrape the text from your business’s PDF Documents using pdftools.
  • How I summarize PDF’s using the OpenAI LLMs in R. This will blow your mind.

SPECIAL ANNOUNCEMENT: AI for Data Scientists Workshop on December 18th

What: GenAI for Data Scientists
When: Wednesday December 18th, 2pm EST
How It Will Help You: Whether you are new to data science or are an expert, Generative AI is changing the game. I’ll show you inside my free Generative AI for Data Scientists workshop.
Price: Does Free sound good?
How To Join: 👉 Register Here

R-Tips Weekly

This article is part of R-Tips Weekly, a weekly video tutorial that shows you step-by-step how to do common R coding tasks.

Businesses are Sitting on $1,000,000 of Dollars of Unstructured Data (and they don’t know how to use it)

Fact: 90% of businesses are not using their unstructured data. It’s true. Many companies have no clue how to extract it. And once they extract it, they have no clue how to use it.

The most common form is text located in PDF documents.

Businesses have 100,000s of PDF documents that contain valuable information.

OpenAI Document Summarization

One of the best use cases of LLMs is document summarization. But how do we get PDF data to OpenAI?

One easy way is in R!

R Tutorial: Scrape PDF Documents and Summarize with OpenAI

This is a simple 2 step process we’ll cover today:

  1. Extract PDF Text: We’ll use pdftools to extract text
  2. Summarize Text with OpenAI’s LLMs: We’ll use httr to connect to OpenAI’s API and summarize our PDF document

Business Objective:

I have set up a PDF document of Meta’s 2024 10K Financial Statement. We’ll use this document to analyze the risks that Meta reported in their filing (without even reading the document).

Load the Libraries

Next, load the libraries. Here’s what we’re using today:

Step 1: Extract PDF Text

With our project set up and libraries loaded, next I’m extracting the PDF text. It’s very easy to do in 1 line of code with pdftools::pdf_text(). Here’s how it works:

Step 2: Summarize the PDF Document with OpenAI LLMs

A common task: I want to know what risks Meta has identified in their 10K Financial Statement.

The solution is to use OpenAI to summarize the document. Here’s the code that does that:

Run this code to send the text and get OpenAI’s response

I’m using httr to send a POST request to OpenAI’s API. Then OpenAI provides a response with the answer to my question in context.

Review the Response

Last, we can review the response from OpenAI’s Chat API. We can see that the top 3 risks are:

  1. Regulatory Compliance
  2. User Privacy and Trust Issues
  3. Competition and Innovation Risks

Conclusions:

You’ve learned my secret 2 step process for PDF Scraping documents and using LLM’s like OpenAI’s Chat API to summarize text data in R.