I am trying to build a scraping script using tor. I have installed tor and firefox on my WSL Ubuntu and I also made sure to uncomment the lines in the torrc file. but when I try to run my code:
from stem import Signal
from stem.control import Controller
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from bs4 import BeautifulSoup
# signal TOR for a new connection
def switchIP():
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)
# get a new selenium webdriver with tor as the proxy
def my_proxy(PROXY_HOST,PROXY_PORT):
fp = webdriver.FirefoxProfile()
# Direct = 0, Manual = 1, PAC = 2, AUTODETECT = 4, SYSTEM = 5
fp.set_preference("network.proxy.type", 1)
fp.set_preference("network.proxy.socks",PROXY_HOST)
fp.set_preference("network.proxy.socks_port",int(PROXY_PORT))
fp.update_preferences()
options = Options()
options.headless = True
return webdriver.Firefox(options=options, firefox_profile=fp)
for x in range(10):
proxy = my_proxy("127.0.0.1", 9050)
proxy.get("https://whatsmyip.com/")
html = proxy.page_source
soup = BeautifulSoup(html, 'lxml')
print(soup.find("span", {"id": "ipv4"}))
print(soup.find("span", {"id": "ipv6"}))
switchIP()
I get the error:
WebDriverException: Message: Reached error page: about:neterror?e=proxyConnectFailure&u=https%3A//whatsmyip.com/&c=UTF-8&d=Firefox%20is%20configured%20to%20use%20a%20proxy%20server%20that%20is%20refusing%20connections.
What am I doing wrong? I tried to connect to https://amazon.com but then it went into an infinite loop.
EDIT: also the connection keeps getting refused if I don't start tor from cmd? is that normal? Can I start tor from the python script itself?