How to stay undetected while scraping the latest post from a social media site that requires login?

user_10568531 0
Viewed 866

I've created a script using python in combination with selenium implementing proxies within it to log in to facebook and scrape the name of the user whose post is on top of my feed. I would like the script to do this every five minutes for an unlimited time.

As this continuous login may lead my account to ban, I thought to implement proxies within the script to do the whole stuff anonymously.

I've written so far:

import random
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def get_first_user(random_proxy):
    options = webdriver.ChromeOptions()
    prefs = {"profile.default_content_setting_values.notifications" : 2}
    options.add_experimental_option("prefs",prefs)
    options.add_argument(f'--proxy-server={random_proxy}')

    with webdriver.Chrome(options=options) as driver:
        wait = WebDriverWait(driver,10)
        driver.get("https://www.facebook.com/")
        driver.find_element_by_id("email").send_keys("username")
        driver.find_element_by_id("pass").send_keys("password",Keys.RETURN)
        user = wait.until(EC.presence_of_element_located((By.XPATH,"//h4[@id][@class][./span[./a]]/span/a"))).text
        return user

if __name__ == '__main__':
    proxies = [`list of proxies`]

    while True:
        random_proxy = proxies.pop(random.randrange(len(proxies)))
        print(get_first_user(random_proxy))
        time.sleep(60000*5)

How to stay undetected while scraping data continuously from a site that requires authentication?

web-scraping selenium python-3.x proxy python
2 Answers

I'm not sure why you would want to continuously login into your Facebook account every 5 minutes to scrape content. And using a random proxy address for each login would likely raise a red flag with Facebook's security rules .

Instead of logging into Facebook every 5 minutes, I would recommend staying logged-in. Selenium has a feature that refreshes a webpage being controlled by automation. By using this method you could refresh your Facebook feed at a predefined interval, such as 5 minutes.

The code below uses this refresh method to reload the page. The code also checks from the user post at the top of your feed.

In testing I noted that Facebook uses some randomized tagging, which is likely used to mitigate scraping. I also noted that Facebook changed the username formats on posts linked to groups, so more testing is required if you want the usernames linked to these posts. I highly recommended conducting more tests to determine what user elements aren't being scraped correctly.


from time import sleep
from random import randint
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException

chrome_options = Options()
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument("--disable-infobars")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-popup-blocking")

# disable the banner "Chrome is being controlled by automated test software"
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation'])

# global driver
driver = webdriver.Chrome('/usr/local/bin/chromedriver', options=chrome_options)

driver.get('https://www.facebook.com')

driver.implicitly_wait(20)

driver.find_element_by_id("email").send_keys("your_username")
driver.find_element_by_id("pass").send_keys("your_password")

driver.implicitly_wait(10)

driver.find_element_by_xpath(("//button[text()='Log In']")).click()


# this function checks for a standard username tag
def user_element_exist():
    try:
        if driver.find_element_by_xpath("//h4[@id][@class][./span[./a]]/span/a"):
            return True
    except NoSuchElementException:
        return False


# this function looks for username linked to Facebook Groups at the top of your feed
def group_element():
    try:
        if driver.find_element_by_xpath("//*[starts-with(@id, 'jsc_c_')]/span[1]/span/span/a/b"):
            poster_name = driver.find_element_by_xpath("//*[starts-with(@id, 'jsc_c_')]/span[1]/span/span/a/b").text
            return poster_name

        if driver.find_element_by_xpath("//*[starts-with(@id, 'jsc_c_')]/strong[1]/span/a/span/span"):
            poster_name = driver.find_element_by_xpath("//*[starts-with(@id, 'jsc_c_')]/strong["
                                                       "1]/span/a/span/span").text
            return poster_name

    except NoSuchElementException:
        return "No user information found"


while True:
    element_exists = user_element_exist()
    if not element_exists:
        user_name = group_element()
        print(user_name)
        driver.refresh()
    elif element_exists:
        user_name = driver.find_element_by_xpath("//h4[@id][@class][./span[./a]]/span/a").text
        print(user_name)
        driver.refresh()

    # set the sleep timer to fit your needs
    sleep(300) # This sleeps for 300 seconds, which is 5 minutes. 

    # I would likely use a random sleep function
    # sleep(randint(180, 360))
user_60834230

Instead of log in every 5 minutes try to move that part away from loop to login only once

if __name__ == '__main__':
    with webdriver.Chrome(options=options) as driver:
        wait = WebDriverWait(driver,10)
        driver.get("https://www.facebook.com/")
        driver.find_element_by_id("email").send_keys("username")
        driver.find_element_by_id("pass").send_keys("password",Keys.RETURN)

        while True:         
            user = wait.until(EC.presence_of_element_located((By.XPATH,"//h4[@id][@class][./span[./a]]/span/a"))).text
            print(user)
            time.sleep(60000*5)

Also consider to use random interval instead of hardcoded sleep:

import random
time.sleep(random.randint(240, 360)) # wait for 4~6 minutes
user_106822890
Related