Issue with ligatures when converting PDF to text in Python (pdfplumber)

userUpUl1 0
Viewed 87

I am running into an issue when trying to convert a PDF to text where the ligatures 'fi' 'ff' 'fl' are being converted to an empty space. I have read through quite a few similar threads on the issue but have not found a solution that works.

This converted text will then be used to match text within a database. So accuracy is paramount.

Link to PDF

fp = 'Inspection_redacted.pdf'

pdf = pdfplumber.open(fp)
fp = fp[:-3] + 'txt'
text_file = open(fp, "w")

for page in pdf.pages:
    text = page.extract_text()
    text_file.write(text)

pdf.close()
text_file.close()

Thanks in advance for any tips!

pdfplumber pdftotext pdf python
2 Answers

pdfplumber seems to not handle ligatures. 'fi', 'ff' and 'fl' are mapped to '\x00' (empty space) unicode mappings. One workaround is to, first, convert the .pdf to an image with pdf2image library then use an OCR tool (e.g, Python-tesseract) to recognize the text embedded in images from the .pdf.

Requirements :

To achieve that, you need to :

  1. Install poppler from here
  2. Install pdf2image: pip install pdf2image
  3. Install tesseract from here
  4. Install pytesseract: pip install pytesseract

Make sure to unzip poppler-0.68.0_x86.7z in C:\Program Files.

Code :

After installing all the requirements needed, you can run the code below :

from pdf2image import convert_from_path
from pytesseract import pytesseract
import os

pytesseract.tesseract_cmd = r'C:\Program Files\Tesseract-OCR\tesseract'

fp = 'Inspection_redacted.pdf'

text_file = open(fp[:-3] + 'txt', "w")

images = convert_from_path(fp, 500, poppler_path=r'C:\Program Files\poppler-0.68.0\bin')

for i, image in enumerate(images):
    fname = 'image'+str(i)+'.png'
    image.save(fname, "PNG")
    text = pytesseract.image_to_string(fname)
    text_file.write(text)
    os.remove(fname)

text_file.close()

>>> Output

@ 7.2.1 FIREPLACES - FIREPLACES (including Gas/LP firelogs) AND CHIMNEYS: Chimney crown/cap cracked

7.2.2 FIREPLACES - FIREPLACES (including Gas/LP firelogs) AND CHIMNEYS: CHIMNEY SWEEP - Excessive
Creosote

O 7.2.3 FIREPLACES - FIREPLACES (including Gas/LP firelogs) AND CHIMNEYS: Cracks - in Firebox

O 7.2.4 FIREPLACES - FIREPLACES (including Gas/LP firelogs) AND CHIMNEYS: Gaps - Seal

© 7.2.5 FIREPLACES - FIREPLACES (including Gas/LP firelogs) AND CHIMNEYS: Chimney-Mortar Joint Gaps

See/Read here the text file in its entirety.

abokey0

TL;DR so busy looking at PDF forgot to test the best extraction see end

I agree OCR can help to locate suspect ligatures, however it is likely that on its own the output may have as many OCR text errors as 14 ligatures you are trying to remove, thus either file compare both outputs for line by line differences (FC.exe or similar helps) or use the OCR fl fi positions to fix the source/output.

If you accept the plain text at face value it is easy to find and replace 14 known culprits in less groupings by dictionary means thus re would likely be fire not flre nor ffre but is possible it is re on its own thus flag that line context for double checking. If you use an editor you may see where correction is needed so now I see I missed an off in my first pass.

enter image description here

Other FnR's should be simpler so ooring is highly probably flooring and under oor unlikely to be any thing other than underfloor

enter image description here

un�nished most likely unfinished (here its easier to see any remaining culprits)

enter image description here enter image description here

That PDF to text is respected in Xpdf and did it well but most users will have the more permissive poppler utils so remove the find filter and redirect outputs in a loop

pdftotext -enc UTF-8 -nopgbrk -layout "path\file.pdf" will output "path\file.txt"

here testing all 14 previously found

poppler-22.04 >library\bin\pdftotext -enc UTF-8 -layout ligatured.pdf -|find  /n "ff"
[23]  3.2.2 PLUMBING SYSTEM - FAUCETS, VALVES AND CONNECTED FIXTURES: Missing shut off handle

poppler-22.04 >library\bin\pdftotext -enc UTF-8 -layout ligatured.pdf -|find  /n "fi"
[56]insulation, air filters, registers): *Asbestos Ducts
[59]  7.2.1 FIREPLACES - FIREPLACES (including Gas/LP firelogs) AND CHIMNEYS: Chimney crown/cap cracked
[61]7.2.2 FIREPLACES - FIREPLACES (including Gas/LP firelogs) AND CHIMNEYS: CHIMNEY SWEEP - Excessive
[63]  7.2.3 FIREPLACES - FIREPLACES (including Gas/LP firelogs) AND CHIMNEYS: Cracks - in Firebox
[64]  7.2.4 FIREPLACES - FIREPLACES (including Gas/LP firelogs) AND CHIMNEYS: Gaps - Seal
[65]  7.2.5 FIREPLACES - FIREPLACES (including Gas/LP firelogs) AND CHIMNEYS: Chimney-Mortar Joint Gaps
[81]  11.2.1 ROOF - ROOF COVERINGS (Surface of roofing materials): Limited Life remaining
[82]  11.2.2 ROOF - ROOF COVERINGS (Surface of roofing materials): Shingle over Wood Shake
[88]13.2.1 INSULATION AND VENTILATION - INSULATION AND VAPOR RETARDERS (in unfinished spaces):
[91]13.2.2 INSULATION AND VENTILATION - INSULATION AND VAPOR RETARDERS (in unfinished spaces):
[94]13.2.3 INSULATION AND VENTILATION - INSULATION AND VAPOR RETARDERS (in unfinished spaces):
[97]13.2.4 INSULATION AND VENTILATION - INSULATION AND VAPOR RETARDERS (in unfinished spaces):

poppler-22.04 >library\bin\pdftotext -enc UTF-8 -layout ligatured.pdf -|find  /n "fl"
[70]  9.2.2 INTERIORS - INTERIORS - General and Visual Mold Assessment : Asbestos - drywall/flooring

enter image description here

userVfkJ0
Related