Python BS4 Table Parsing Returning Empty List

Question

I'm trying to parse a table of proxies from the following link https://www.proxy-list.download/SOCKS5 Using inspect element I can see that the 'tbody' element has 'tr' elements in it and 'td' elements within those.

However, when I try to access these and pull the proxies from each row and column it's as if they don't exist.

It just returns an empty list.

I've tried switching the lxml parsing to html, and accessing the tbody without it's class. I'm just not quite sure why I can print the 'soup' variable and get this output

but not get an output from the rows, etc.

import requests
from bs4 import BeautifulSoup, NavigableString
from selenium import webdriver

#While Loop Control Variable to Make Sure We Get a Working Proxy
successful_access = False

#Pulls the HTML of Our Proxy Website
website_url = requests.get("https://www.proxy-list.download/SOCKS5")
#Makes a Soup Variable That Turns Raw HTML Into "Parse-able" Data
soup = BeautifulSoup(website_url.content,"lxml")
#Puts The Table Object Into The "table" Variable
table = soup.find('tbody', attrs={'class':'table-hover'})
rows = table.find_all('tr')
print(rows)

#Initializes an Empty List for All Proxies
proxy_list = []

#For Each Row in Our List of Rows, We Extract The IP Number and Port, Appending Them To Our List Of IPs
for row in rows:
    cols = rows.find_all('td')
    ip = cols[0]
    port = cols[1]
    fullip = ip + ":" + port
    proxy_list.append(fullip)

There is no output for the

print(rows)

line, and naturally, the "proxy_list" list is empty as well.

[]

user_6241235 · Answer

Use the API endpoint returning json which feeds that page. You can find it in the network tab of browser when refreshing the url you supplied.

import requests

r = requests.get('https://www.proxy-list.download/api/v0/get?l=en&t=socks5').json()
print(r[0]['LISTA'])

I like this format but you could use pandas to spit out a dataframe

import requests
import pandas as pd

r = requests.get('https://www.proxy-list.download/api/v0/get?l=en&t=socks5').json()
df = pd.io.json.json_normalize(r[0]['LISTA'])
print(df)

Python BS4 Table Parsing Returning Empty List

1 Answers