Only scrape paragraphs containing certain words in PDF embedded URLs

user_14843068 0
Viewed 115

I'm currently developing some code to scrape text from websites. I'm not interested to scrape the entire page, but just in sections of the page that contain certain words. I've managed to do so for most URLs using the .find_all("p") command, however this does not work for URLs that are directed to a PDF.

I cannot seem to find a way to open a PDF as a text and then divide the text into paragraphs. This is what I would like to do: first 1) Open a PDF embedded URL as a text, and 2) Divide this text into multiple paragraphs. This way, I can scrape only paragraphs containing certain words.

Below is the code I'm currently using to scrape paragraphs containing certain words for "normal" URLs. Any tips to make this work for PDF embedded URLs (such as the variable 'url2', code below) is much appreciated!

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
import re

url1 = "https://brainybackpackers.com/best-places-for-whale-watching-in-the-world/"
url2 = "https://www.environment.gov.au/system/files/resources/7f15bfc1-ed3d-40b6-a177-c81349028ef6/files/aust-national-guidelines-whale-dolphin-watching-2017.pdf"
url = url1
req = Request(url, headers={"User-Agent": 'Mozilla/5.0'})
page = urlopen(req, timeout = 5) # Open page within 5 seconds. This line skips 'empty' websites
htmlParse = BeautifulSoup(page.read(), 'lxml') 
SearchWords = ["orca", "killer whale", "humpback"] # text must contain these words

# Check if the article text mentions the SearchWord(s). If so, continue the analysis. 
if any(word in htmlParse.text for word in SearchWords):
    textP = ""
    text = ""
    
    # Look for paragraphs ("p") that contain a SearchWord
    for word in SearchWords:
        print(word)
        for para in htmlParse.find_all("p", text = re.compile(word)): 
            textParagraph = para.get_text()
            textP = textP + textParagraph
    text= text + textP
    print(text)
web-scraping nlp pdf python
2 Answers

One thing you could try is the pdfminer.six package. After importing this, we can utilize the pdfminer.high_level.extract_text() function. By importing it, we can scrape a pdf:

import pdfminer.high_level as pdfminer

infile = "my/file/path.pdf" # file you want to turn into text

out_text = pdfminer.extract_text(infile) # extract the text to out_file var

# out_text now contains a string of your pdf contents

It should be noted that the extract_text function works on a local file, so we'll need to save the pdf to some local buffer that you can delete at a later time. If you're on a Unix-Like Operating System, I'd say something like /tmp/.

Moving to your implementation, I believe you'd want something like this:

import pdfminer.high_level as pdfminer
import requests

# get the pdf and save it
url = "https://www.environment.gov.au/system/files/resources/7f15bfc1-ed3d-40b6-a177-c81349028ef6/files/aust-national-guidelines-whale-dolphin-watching-2017.pdf"
response = requests.get(url)
pdf_name = url.split('/')[-1] # everything right of the last slash
pdf_path = "/tmp/" + pdf_name # CHANGE TO WHATEVER "BUFFER" FOLDER YOU WANT

# save the pdf locally to be used with the pdf parser
with open(pdf_path,'wb') as outfile:
    outfile.write(response.content)

# read the contents of the pdf into the out_text var
out_text = pdfminer.extract_text(pdf_path)

# out_text now contains a string of your pdf contents

From here you should be free to scrape all you want.

user_157861920

You could read the PDF and search pages for the contents you are after:

# pip install pyPDF2

import io
import requests
import PyPDF2


URI = "https://www.environment.gov.au/system/files/resources/7f15bfc1-ed3d-40b6-a177-c81349028ef6/files/aust-national-guidelines-whale-dolphin-watching-2017.pdf"

r = requests.get(URI)
with io.BytesIO(r.content) as f:
  reader = PyPDF2.PdfFileReader(f)
  num_pages = reader.numPages
  
  data = []
  # place page text to data
  for page in range(num_pages):
    page_data = reader.getPage(page)
    data.append(page_data.extractText())

# look up
search_words = set(["orca", "killer whale", "humpback"])

# get pages containing your lookup
wanted_page = []
for page_contents in data:
     for word in search_words:
         if word in page_contents.lower():
             wanted_page.append(page_contents)

             
print(wanted_page)
user_68582440
Related