Save scraped list object text as column to pandas dataframe

user_14320576 0
Viewed 82

I want to scrape text from webpages and put it in to pandas dataframe. If I can scrape a table I get no problem but this here is no table and make me much trouble.

driver = webdriver.Firefox()

driver.get('https://example.com/')
time.sleep(3)

number = driver.find_elements_by_xpath("//span[@class='blaaal']")
name = driver.find_elements_by_xpath("//span[@class='blaaal2']")
for count in number:
    print(count.text)

for names in name:
    print(names.text)

df = pd.DataFrame(columns=['name','number'])

In that way I can print the data but if I use the count names definition without for in loop in dataframe I get error list object got no text or something like that.

I don't know how to put it to dataframe I think I have to append to define the for in loop as a column right? I don't find any post here on stack or any other learn tutorials on google.

pandas selenium-webdriver dataframe selenium python
1 Answers

Create list first for both number and name and then pass into pandas.

number =[count.text for count in driver.find_elements_by_xpath("//span[@class='blaaal']")]
name = [names.text for names in driver.find_elements_by_xpath("//span[@class='blaaal2']")]

df = pd.DataFrame({"name":name,"number":number})
user_108856840
Related