scraping select all checkbox using python

user_16029488 0
Viewed 89
import requests
from bs4 import BeautifulSoup
import pandas as pd
url =  "https://www.pivottrading.co.in/beta/tools/open-high-low-scanner.php?broker=zerodha"
page = requests.get(url)
soup = BeautifulSoup(page.text, 'lxml')
table = soup.find('table', {'class' : 'table'})
rows = table.find_all('th')

headers = []
for i in table.find_all('th'):
    title = i.text
    headers.append(title)

df = pd.DataFrame(columns = headers)
for row in table.find_all('tr')[1:]:
        data = row.find_all('td')
        row_data = [td.text.strip() for td in data]
        length = len(df)
        df.loc[length] = row_data
print(df)

I need to scrape a table from a website but it has select all checkbox for each row .What should I do. Any help will be appreciated thank you.

pandas beautifulsoup python
2 Answers

(If I understand your question correctly: you want to remove the checkboxes from the output of the table).

Since the checkboxes are the first index of the table, you can skip them using index slicing. Use: [1:], which means: "from the first index to the last" (skipping the zero-based index).

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = (
    "https://www.pivottrading.co.in/beta/tools/open-high-low-scanner.php?broker=zerodha"
)
page = requests.get(url)
soup = BeautifulSoup(page.text, "lxml")
table = soup.find("table", {"class": "table"})
rows = table.find_all("th")

headers = []
for i in table.find_all("th"):
    title = i.text.strip()
    headers.append(title)

rows = []

for row in table.find_all("tr")[1:]:
    data = row.find_all("td")
    rows.append(td.text.strip() for td in data[1:])

df = pd.DataFrame(rows[:-1], columns=headers[1:])

print(df)

Output:

       Scrip  P.Close     Open     High      Low LTP @ REAL LTP(NOW) Result
0  BRITANNIA  3379.10  3385.00  3447.00  3385.00    3439.50  3439.50      0
1  EICHERMOT  2551.20  2565.00  2634.00  2565.00    2625.05  2625.05      0
user_123497340

  1. You don't need to check those boxes in order to return all rows.

  2. You can grab the table with pandas and drop the first column by name (if desired).

  3. You can also do some tidying to match the web page.

import pandas as pd

df = pd.read_html('https://www.pivottrading.co.in/beta/tools/open-high-low-scanner.php?broker=zerodha')[0]
df.drop(columns={'Sr.No.'}, inplace=True)
df.iloc[-1, 0:4] = ''
df.fillna(0, inplace=True)
df
user_62412350
Related