page_source in selenium changes

EUN WOO LEE 0
Viewed 18

I'm trying to make crawler for Youtube. I encountered strange behavior. In the following source code, driver.page_source is obtained by selenium. I passed the result to Beautifulsoup for parsing. The problem is that the length of driver.page_source changes. How can this happen? Is there any idea about this?

elif 'src' in seq:
    print('video-src')
    print(seq['src'])
    soup = bs(driver.page_source, "html.parser")
    print('driver.page_source length='+str(len(driver.page_source)))
    f = open('test.txt','w',encoding='UTF-8')
    f.write(driver.page_source)
    f.close()
    print('driver.page_source length='+str(len(driver.page_source)))
    tag = '<span dir="auto" class="style-scope yt-formatted-string">'
    find_start = driver.page_source.find(tag+'댓글')
    print('driver.page_source length='+str(len(driver.page_source)))
    tag_value = driver.page_source[find_start:find_start+200]                            
    print('driver.page_source length='+str(len(driver.page_source)))
    p = re.compile('\>([\d,]+)\<')
    m = p.search(tag_value)
    if m:
        print(m.group(1))
        video[item['name']] = m.group(1)
    else:
        print('error')
        print(tag_value)

driver.page_source length=4103114
driver.page_source length=4102392
driver.page_source length=4102392
driver.page_source length=4103129
selenium
1 Answers

The page_source can change, elements can be loaded later.
Instead of checking the page_source length you can save the different driver.page_sources in text files and compare them to understand what is different. A method to do so could be using difflib:

import difflib

source1 = driver.page_source
file1 = open("file1.txt", "w")
text1 = file1.write(source1)
text1.close()

source2 = driver.page_source
file2 = open("file2.txt", "w")
text2 = file2.write(source2)
text2.close()

with open('file1.txt') as file_1:
    file_1_text = file_1.readlines()
 
with open('file2.txt') as file_2:
    file_2_text = file_2.readlines()
 
# Find and print the diff:
for line in difflib.unified_diff(
        file_1_text, file_2_text, fromfile='file1.txt',
        tofile='file2.txt', lineterm=''):
    print(line)
Carapace0
Related