Appending table values from a webpage to csv

user_778942 0
Viewed 154

I want to get the table from webpage

import os
from webdriver_manager.chrome import ChromeDriverManager
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--start-maximized')
options.page_load_strategy = 'eager'
options.add_argument("--headless");
driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 20)   
driver.get("https://munafasutra.com/nse/dividends")
file_object = open('divident.csv', 'a')

output table

enter image description here

How to get the first table and their values?

webdriver selenium python-3.x python
3 Answers

You can use below XPATH to retrieve the first table value :

//h3[text()=' Earlier dividends announced by companies ']/preceding-sibling::table/descendant::td

Something like this :

driver.get("https://munafasutra.com/nse/dividends")
first_table = driver.find_elements(By.XPATH, "//h3[text()=' Earlier dividends announced by companies ']/preceding-sibling::table/descendant::td")
for first in first_table: 
   print(first.text)
user_94440730

You have to look at the HTML path and locate the WebElement that is gathering that first table (Clicking "Inspect" when you click right-button of the mouse can do the work).

You can save that webelement using the following line of code:

first_table = driver.find_element_by_xpath("//div[@id = 'co']//table[1]") # The [1] is not really necessary as when using **find_element_by_xpath** will only look for the first element.

Then, if you look at how data is organized inside that table, you can observe each row is gathered by a tr WebElement. Therefore, if you wish to write it in a csv file, I would suggest to write row by row with the following code:

rows = first_table.find_elements_by_xpath("./tbody/tr") 
for row in rows:
    entries_of_the_row = row.find_elements_by_xpath("./td") 
    row_to_csv = []
    for entry in entries_of_the_row:
        row_to_csv.append(entry.text)
    file_object.write(f"{row_to_csv[0]}, {row_to_csv[1]}, {row_to_csv[2]}, {row_to_csv[3]}, {row_to_csv[4]}\n")   
file_object.close()
user_157042810

You can use BeautifulSoup to get the table data. Selenium is not required if you just want to extract web page data.

You need to import below packages :

import csv
from urllib.request import urlopen
from bs4 import BeautifulSoup

You can Extract HTML of the table using below code (soup variable will contain the HTML code of the entire page):

url_munafasutra = "https://munafasutra.com/nse/dividends"
html_munafasutra = urlopen(url_munafasutra)
soup = BeautifulSoup(html_munafasutra, 'html')

Below is the code to extract HTML for the 1st table (here table is the tag value and in [] contains the index of the table we want to extract data of):

first_table = soup.find_all('table')[0]

You can also add attributes to distinctly identify the table along with tag name.

Below is the code to extract all the rows in the selected table :

all_rows = first_table.findAll("tr")

Use the below code to write the data in csv file :

with open("C:\\Users\\abhay\\.spyder-py3\\table_extract.csv", "wt+", newline="") as f:
    table_to_csv = csv.writer(f)
    for row in all_rows:
        row_data = []
        for cell in row.findAll(["td", "th"]):
            row_data.append(cell.get_text())
        table_to_csv.writerow(row_data)

Below is the complete code to extract 1st table data to csv :

import csv
from urllib.request import urlopen
from bs4 import BeautifulSoup

url_munafasutra = "https://munafasutra.com/nse/dividends" 
html_munafasutra = urlopen(url_munafasutra)
soup = BeautifulSoup(html_munafasutra, 'html')

first_table = soup.find_all('table')[0]
all_rows = first_table.findAll("tr")

with open("C:\\Users\\abhay\\.spyder-py3\\table_extract.csv", "wt+", newline="") as f:
    table_to_csv = csv.writer(f)
    for row in all_rows:
        row_data = []
        for cell in row.findAll(["td", "th"]):
            row_data.append(cell.get_text())
        table_to_csv.writerow(row_data)
user_101444480
Related