Parsing leetcode question content with requests and BeautifulSoup

user_8836582 0
Viewed 1696

I am trying to parse the content of the interview questions on Leetcode.

For example, on https://leetcode.com/problems/two-sum/,

I am trying to get 

Given an array of integers, return indices of the two numbers such that they add up to a specific target.

You may assume that each input would have exactly one solution, and you may not use the same element twice.

It didn't seem that hard. I used requests and BeautifulSoup to do it:

    url = 'https://leetcode.com/graphql/two-sum'
    try:
        page = requests.get(url)
    except (requests.exceptions.ReadTimeout,requests.exceptions.ConnectTimeout):
        print('time out')
        return 'time out'

    soup = BeautifulSoup(page.content, 'html.parser')
    print(soup.prettify())

However, as you can see in the response of the page on the page through developer console (F12), the response does not include the content shown on the page.

Is there a way to get this content?

python-requests web-scraping beautifulsoup python
3 Answers

You don't need selenium. The page does a POST request for dynamic content. Basically, sends a MySql query to a backend database. So, it is much quicker to do the following:

import requests
from bs4 import BeautifulSoup as bs

data = {"operationName":"questionData","variables":{"titleSlug":"two-sum"},"query":"query questionData($titleSlug: String!) {\n  question(titleSlug: $titleSlug) {\n    questionId\n    questionFrontendId\n    boundTopicId\n    title\n    titleSlug\n    content\n    translatedTitle\n    translatedContent\n    isPaidOnly\n    difficulty\n    likes\n    dislikes\n    isLiked\n    similarQuestions\n    contributors {\n      username\n      profileUrl\n      avatarUrl\n      __typename\n    }\n    langToValidPlayground\n    topicTags {\n      name\n      slug\n      translatedName\n      __typename\n    }\n    companyTagStats\n    codeSnippets {\n      lang\n      langSlug\n      code\n      __typename\n    }\n    stats\n    hints\n    solution {\n      id\n      canSeeDetail\n      __typename\n    }\n    status\n    sampleTestCase\n    metaData\n    judgerAvailable\n    judgeType\n    mysqlSchemas\n    enableRunCode\n    enableTestMode\n    envInfo\n    libraryUrl\n    __typename\n  }\n}\n"}

r = requests.post('https://leetcode.com/graphql', json = data).json()
soup = bs(r['data']['question']['content'], 'lxml')
title = r['data']['question']['title']
question =  soup.get_text().replace('\n',' ')
print(title, '\n', question)
user_62412350

You need to load java scripts in page and then get page content. The easiest way to do this is by using Selenium.

from selenium import webdriver
from time import sleep
import os


# initialise browser
browser = webdriver.Chrome(os.getcwd() + '/chromedriver')
# load page
browser.get('https://leetcode.com/problems/two-sum/')

# execute java script
browser.execute_script("return document.getElementsByTagName('html')[0].innerHTML")

# wait page to load
sleep(5)

# get selected content
problem_description = browser.find_element_by_class_name('question-content__JfgR')
print(problem_description.text)

Output:

Given an array of integers, return indices of the two numbers such that they add up to a specific target.
You may assume that each input would have exactly one solution, and you may not use the same element twice.
Example:
Given nums = [2, 7, 11, 15], target = 9,

Because nums[0] + nums[1] = 2 + 7 = 9,
return [0, 1].
user_115579740

The website is generated by executing dynamic Javascript. So you won't get it simply using requests. You could use selenium to simulate a Firefox browser.

Check the tutorial.

user_65919900
Related