I need to scrape information from this page: https://professionals.tarkett.co.uk/en_GB/collection-C001030-arcade/arcade-b023-2128.
There are 6000+ of those pages I need to scrape. I really don't want to use selenium as it is too slow for this type of job.
The information I am trying to scrape is the 'Document' section down the bottom page, just above the 'Case studies' and 'About' section. There is a pdf DataSheet and several other key bits on information in that area I need to scrape but am finding absolutely impossible.
I have tried everything at this point, requests, dryscrape, requests_html etc. and nothing works.
It seems the information I need is being rendered by JavaScript. I have tried using libraries that supposedly work for these type of issues but in my case it's not working.
Here's snippet of code to show:
from requests_html import HTMLSession
from bs4 import BeautifulSoup as bs
session = HTMLSession()
resp = session.get("https://professionals.tarkett.co.uk/en_GB/collection-C001030-arcade/arcade-b023-2128", headers=headers) # tried without headers too
resp.html.render()
soup = bs(resp.html.html, "html.parser")
soup.find("section", {"id" : "collection-documentation"})
Output:
<section data-v-6f916884="" data-v-aed8933c="" id="collection-documentation"><!-- --></section>
No matter what I try, the information just isn't there. This is one element specifically that I am trying to get:
<a data-v-5a3c0164="" href="https://media.tarkett-image.com/docs/DS_INT_Arcade.pdf" target="_blank" rel="noopener noreferrer" class="basic-clickable tksb-secondary-link-with-large-icon" is-white-icon="true"><svg xmlns="http://www.w3.org/2000/svg" width="47" height="47" viewBox="0 0 47 47" class="tksb-secondary-link-with-large-icon__icon"><g transform="translate(-1180 -560)"><path d="M1203.5,560a23.5,23.5,0,1,1-23.5,23.5A23.473,23.473,0,0,1,1203.5,560Z" class="download-icon__background"></path></g> <g><path d="M29.5,22.2l-5.1,5.5V10.3H22.6V27.7l-5.1-5.5-1.4,1.2,7.4,7.9,7.4-7.9Z" class="download-icon__arrow-fill"></path> <g><path d="M31.6,37.6H15.4V31.3h1.8v4.5H29.8V31.3h1.8Z" class="download-icon__arrow-fill"></path></g></g></svg> <span class="tksb-secondary-link-with-large-icon__text-container"><span class="tksb-secondary-link-with-large-icon__label">Datasheet</span> <span class="tksb-secondary-link-with-large-icon__description">PDF</span></span></a>
The best I've come up with so far is finding this URL from using Chrome Dev tools and inspecting the network tab to see what happens when I scroll into view of the data I want; https://professionals.tarkett.co.uk/en_GB/collection-product-formats-json/fb02/C001030/b023-2128?fields[]=sku_design&fields[]=sku_design_key&fields[]=sku_thumbnail&fields[]=sku_hex_color_code&fields[]=sku_color_family&fields[]=sku_delivery_sla&fields[]=sku_is_new&fields[]=sku_part_number&fields[]=sku_sap_number&fields[]=sku_id&fields[]=sku_format_type&fields[]=sku_format_shape&fields[]=sku_format&fields[]=sku_backing&fields[]=sku_items_per_box&fields[]=sku_surface_per_box&fields[]=sku_box_per_pallet&fields[]=sku_packing_unit_code&fields[]=sku_collection_names&fields[]=sku_category_b2b_names&fields[]=sku_sap_sales_org&fields[]=sku_minimum_order_qty&fields[]=sku_base_unit_sap&fields[]=sku_selling_units&fields[]=sku_pim_prices&fields[]=sku_retailers_prices&fields[]=sku_retailers_prices_unit&fields[]=sku_installation_method.
Now I could easily scrape the information I want from here (which at this rate I will probably have to) as it does have key information in there I need that isn't being loaded from HTML. So all I'll have to do is extract each products ID code and modify each URL to do it. But even then, the ONLY but of information this still doesn't have is the DataSheet URL. I thought I had figured it all out when I discovered this but no, this still leaves me stuck in the mud and I am sinking fast trying to find any solution other than selenium to extracting this one bit of info in a terminal using requests and libraries alike. I'm implementing threading as well which is why it's really important for me to be able to do this without loading up a browser like selenium.
Is it even possible at this point? I'd really appreciate someone who actually knows what they're talking unlike me, to take a look at the page, tell me what I'm missing or point me in the right direction. I need this finished by today and I've been pressed on this for 2 days now and I am starting to give up.