Trouble fetching ports from a webpage using requests

Question

I'm trying to parse list of proxies from a webpage. The script that I've written can fetch the ips from that site. However, the port appears to be dynamic and the script can't grab them. I suppose there is any way out there to grab the port as well using requests module as I'm not interested to use any browser simulator.

webpage address

import requests
from bs4 import BeautifulSoup

link = "https://premproxy.com/list/"

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36'
    res = s.get(link)
    soup = BeautifulSoup(res.text,"lxml")
    for item in soup.select("table#proxylistt > tbody > tr.anon"):
        proxy = item.select_one("td[data-label^='IP:port']").text
        print(proxy)

Current output are like:

52.194.230.130:
37.120.192.154:
203.202.245.58:
190.211.105.86:

Expected output:

52.194.230.130:8080
37.120.192.154:8080
203.202.245.58:80
190.211.105.86:55443

How can I scrape ports from that site using requests?

user_6241235 · Answer

The ports, as you noted, are dynamically added.

If you look at the view source, and inspect the table, you will notice the following class mapping for the value attributes e.g. value="138.94.255.227|r0f21"; associated with the proxies:

138.94.255.227:

The mappings for the class to port e.g. r0f21 to port 80, are held in this packed js file:

https://premproxy.com/js/04436.js

However, that file changes its name over time, so you need a way to pick that up dynamically:

'https://premproxy.com' + soup.select_one('script[src*="/js/"]')['src']

Currently, if I unpack that file, I get:

$(document).ready(function()
    {
    $('.rb88c').html(8080);
    $('.r7671').html(53281);
    $('.rc88a').html(42860);
    # ............ other values
    $('.re376').html(47504)
}
);

So, in theory, if you can work out how to dynamically unpack that js file, you can easily create a lookup/dictionary to apply to the current response you are getting from the initial url.

This SO Q&A identifies a suitable library though the answers there are not suitable in this case, so I had a look at the documentation and put in the relevant method call below.

Working code:

from bs4 import BeautifulSoup as bs
import jsbeautifier
import re, requests

with requests.Session() as s:
    s.headers = {'User-Agent':'Mozilla/5.0'}
    r1 = s.get('https://premproxy.com/list/').text
    soup = bs(r1, 'lxml')
    link = 'https://premproxy.com' + soup.select_one('script[src*="/js/"]')['src']
    r2 = s.get(link).text
    unpack = jsbeautifier.beautify(r2)
    p = re.compile(r"'\.(.*?)'\)\.html$(\d+)$", re.M)
    lookup = {i[0].replace('\',''):i[1] for i in  p.findall(unpack)}
    proxies = [i['value'].split('|') for i in soup.select('#proxylistt [name="proxyIp[]"]')]
    result = {i[0]:lookup[i[1]] for i in proxies if i[1] in lookup}
    print(result)

Trouble fetching ports from a webpage using requests

1 Answers