Trying to update code to return HTML value in Selenium (Python)

user_19120926 0
Viewed 43

I am using Selenium to crawl some Facebook group information:

    with open("groups.txt") as file:
        lines = file.readlines()
    total = len(lines)
    count = 1
    for line in lines:

        group_id = line.strip().split(".com/")[1]
        if "groups" not in line:
            new_line = "https://www.facebook.com/groups/" + str(group_id) + "/about"
        else:
            new_line = line.strip() + '/about'
        sleep(2)
        driver.get(new_line)
        page_source = driver.page_source
        page_id = page_source.split('"groupID":"')[1].split('","')[0]
        page_followers = page_source.split('<!-- --> total members')[0][-15:]
        page_followers = str(page_followers.split('>')[1]).replace(',', '')
        page_name = page_source.split("</title>")[0].split("<title>")[1]

        df1.loc[len(df1)] = [line.strip(), 'https://www.facebook.com/' + str(page_id), page_followers, page_name]
        print(f"{count}/{total}", line.strip(), 'https://www.facebook.com/' + str(page_id), page_followers)
        count += 1
    df1.to_csv("groups.csv", encoding='utf-8', index=False, header=False)

Facebook has updated something recently so this code fails to return number of group members.

These are the relevant lines:

page_followers = page_source.split('<!-- --> total members')[0][-15:]
page_followers = str(page_followers.split('>')[1]).replace(',', '')

Taking view-source:https://www.facebook.com/groups/764385144252353/about as an example, I find two instances of "total members". Is it possible to get some advice on what I should change to be able to catch this number?

selenium html python
1 Answers

NEW

This code extract the exact number of members and convert it from string to integer

driver.get('https://www.facebook.com/groups/410943193806268/about')
members = driver.find_element(By.XPATH, "//span[contains(text(), 'total members')]").text
members = int(''.join(i for i in members if i.isdigit()))
print(members)

output

15589

OLD

I suggest don't use page_source to extract this kind of data, instead use find_element in this way

driver.find_element(By.CSS_SELECTOR, "a[href*='members']").text.split()[0]

output

'186'

Explanation: a[href*='members'] search for a elements (for example <a class='test'>...</a>) having a href attribute containing the string members (for example <a href="something-members-test">...</a>)

user_81573040
Related