I am having some trouble extracting a specific value from an element within the attributes extracted from a website using the code below:
from bs4 import BeautifulSoup
import requests
# Get mills and estates information from dashboard
url = 'http://nestetraceabilitydashboard.com/nestes-palm-oil-dashboard'
page = requests.get(url).text
soup = BeautifulSoup(page, "html.parser")
divList = soup.findAll('div', attrs={"class" : "map-item estate-map-item"})
data = {}
for div in divList:
for k,v in div.attrs.items():
if k not in ('class'):
data[k] = data.get(k, []) + [v]
df = pd.DataFrame(data)
An excerpt of the divList is as below:
[<div class="map-item estate-map-item" data-country="Indonesia" data-latitude="1.926944000" data-location="Riau" data-longitude="99.906390000" data-mills="Aek Nabara" id="map_item_5600">(Aek Nabara) - Aek Nabara</div>,
<div class="map-item estate-map-item" data-country="Indonesia" data-latitude="0.429444444" data-location="Riau" data-longitude="101.818611100" data-mills="Buatan I " id="map_item_5601">(Buatan I/II ) - Buatan</div>,
However, the output dict and dataframe removes everything after the map_item_XXXX in the id.
How would I go about getting the values only outside the quotation marks in my dict and subsequently into thedataframe id column e.g (Aek Nabara) - Aek Nabara for the first item in the divList above?