PyPDF4 not reading certain characters

Viewed 222

I'm compiling some data for a project and I've been using PyPDF4 to read this data from it's source PDF file, but I've been having trouble with certain characters not showing up correctly. Here's my code:

from PyPDF4 import PdfFileReader
import pandas as pd
import numpy as np
import os
import xml.etree.cElementTree as ET

# File name
pdf_path = "PV-9-2020-10-23-RCV_FR.pdf"

# Results storage
results = {}

# Start page
page = 5

# Lambda to assign votes
serify = lambda voters, vote: pd.Series({voter.strip(): vote for voter in voters})

with open(pdf_path, 'rb') as f:
    # Get PDF reader for PDF file f
    pdf = PdfFileReader(f)

    while page < pdf.numPages:
        # Get text of page in PDF
        text = pdf.getPage(page).extractText()

        proposal = text.split("\n+\n")[0].split("\n")[3]

        # Collect all pages relevant pages
        while text.find("\n0\n") is -1:
            page += 1
            text += "\n".join(pdf.getPage(page).extractText().split("\n")[3:])
        # Remove corrections
        text, corrections = text.split("CORRECCIONES")
        
        # Grab relevant text !!! This is where the missing characters show up.
        text = "\n, ".join([n[:n.rindex("\n")] for n in text.split("\n:")])
        for_list = "".join(text[text.index("\n+\n")+3:text.index("\n-\n")].split("\n")[:-1]).split(", ")
        nay_list = "".join(text[text.index("\n-\n")+3:text.index("\n0\n")].split("\n")[:-1]).split(", ")
        abs_list = "".join(text[text.index("\n0\n")+3:].split("\n")[:-1]).split(", ")
        # Store data in results
        results.update({proposal: dict(pd.concat([serify(for_list, 1), serify(nay_list, -1), serify(abs_list, 0)]).items())})
        page += 1
        print(page)
results = pd.DataFrame(results)

The characters I'm having difficulty don't show up in the text extracted using extractText. Ždanoka for instance becomes "danoka, Štefanec becomes -tefanc. It seems like most of the characters are Eastern European, which makes me think I need one of the latin decoders.

I've looked through some of PyPDF4's capabilities, it seems like it has plenty of relevant codecs, including latin1. I've attempted decoding the file using different functions from the PyPDF4.generic.codecs module, and either the characters don't show still, or the code throws an error at an unrecognised byte.

I haven't yet attempted using multiple codecs on different bytes from the same file, that seems like it would take some time. Am I missing something in my code that can easily fix this? Or is it more likely I will have to tailor fit a solution using PyPDF4's functions?

0 Answers
Related