pyinstaller - Question regarding Scrape Project

Question

This is my first true Python project. After spending about 40 hours on it, I got it to work as intended.

The two main features are:

Access Airtable
Scrape Google Shopping

It grabs a URL from Airtable, scrapes Google shopping prices and then puts them into a different Airtable. It uses requests-html to do this. When I try to compile it with pyinstaller I get the following errors when running the exe:

C:\Users\kibit\PycharmProjects\AirtableGoogleShopping\dist\main>main
Traceback (most recent call last):
  File "main.py", line 2, in 
  File "", line 1007, in _find_and_load
  File "", line 986, in _find_and_load_unlocked
  File "", line 680, in _load_unlocked
  File "PyInstaller\loader\pyimod03_importers.py", line 546, in exec_module
  File "requests_html.py", line 9, in 
  File "", line 1007, in _find_and_load
  File "", line 986, in _find_and_load_unlocked
  File "", line 680, in _load_unlocked
  File "PyInstaller\loader\pyimod03_importers.py", line 546, in exec_module
  File "pyppeteer\__init__.py", line 43, in 
NameError: name '__version__' is not defined
[8144] Failed to execute script 'main' due to unhandled exception!

Right now I believe it maybe has something to do, that requests-html is using chromium for scraping and I have to somehow add this to pyinstaller. Then again I really don't have a correct idea. Maybe something else is missing, which I somehow have to point pyinstaller to?

Here is my main.py. Probably a bit messy, sorry!

import requests
from requests_html import HTMLSession
import json
import re
from pygame import mixer
import time

mixer.init()
sound=mixer.Sound("bell.wav")

AIRTABLE_BASE_ID = "***"
AIRTABLE_API_KEY = "***"
AIRTABLE_TABLE_NAME = "***"
AIRTABLE_TABLE_NAME_URL = "***"

endpoint = f'https://api.airtable.com/v0/{AIRTABLE_BASE_ID}/{AIRTABLE_TABLE_NAME}'
endpoint_url = f'https://api.airtable.com/v0/{AIRTABLE_BASE_ID}/{AIRTABLE_TABLE_NAME_URL}'


def add_to_airtable(produkt_air, retailer_air, preis_air, url_air):
    headers = {
        "Authorization": f"Bearer {AIRTABLE_API_KEY}",
        "Content-Type": "application/json"
    }
    data = {
        "records": [
            {
                "fields": {
                    "Produkt": produkt_air,
                    "Retailer": retailer_air,
                    "Preis": preis_air,
                    "Url": url_air
                }
            }
        ]
    }
    r = requests.post(endpoint, json=data, headers=headers)
    return r.status_code == 200


session = HTMLSession()

headers = {
        "Authorization": f"Bearer {AIRTABLE_API_KEY}",
        "Content-Type": "application/json"
    }

r_get = requests.get(endpoint_url, headers=headers)

table_dict = json.loads(r_get.text)

for tbl in table_dict['records']:
    url = (tbl['fields']['Url'])  # Url ist die Spalte die gelesen wird
    print(url)
    r = session.get(url)

    produktscan = r.html.find('title')
    produkt = produktscan[0].text
    produkt = produkt.replace(' | Google Shopping', '')

    print(produkt)

    for article in r.html.find('tr.sh-osd__offer-row'):
        retailer = article.find('div.kPMwsc', first=True).full_text
        retailer = retailer.replace('Wird in einem neuen Fenster geöffnet', '')
#       preis = article.find('div.drzWO', first=True).text  #alte Suchvariable die manchmal nicht funktioniert
        try:
            preis1 = article.find('span.g9WBQb.fObmGc', first=True).text
            preis2 = preis1.replace('’', '')
            preis3 = re.findall("([0-9]+[,.]+[0-9]+)", preis2)
            preis = float(preis3[0])
        except:
            preis=0
        add_to_airtable(produkt_air=produkt, retailer_air=retailer, preis_air=preis, url_air=url)

        print(retailer)
        print(preis)

sound.play()
time.sleep(1)

pyinstaller - Question regarding Scrape Project

0 Answers