How to access data of a webpage using Django?

user_14866603 0
Viewed 67

I am new to Django. I am trying to access this webpage's data and store it in some database (like sqlite). But I couldn't fetch the data. I am not sure whether this task requires the use of IMDB API or if it's possible without the use of it.

This is what I have tried:

# view.py

from django.shortcuts import render
import requests 
from django.http import HttpResponse
import urllib.request, json 

def index(request):
    r = requests.get('https://www.imdb.com/chart/top?ref_=nv_mv_250').json()
    d = {'v': r}
    return render(request, 'index.html', context = d)

I am getting this error by running the above code:

JSONDecodeError at /
Expecting value: line 4 column 1 (char 3)

It would be great if someone can help me out.

django-rest-framework rest django python json
2 Answers

The problem here is that, the webpage you're trying to access returns HTML, and not any type of HttpResponse a normal api would return. The JSON decode error occurred because your script was basically trying to decode the raw html page. Here are a couple of solutions to your problem:

user_138924760

As @CosmicReindeer pointed out, you can use Beautiful Soup in python. You can view the class of title, year and rating by viewing the page source. And don't forget, the data is in the form of table.

from bs4 import BeautifulSoup
import requests

page = requests.get("https://www.imdb.com/chart/top?ref_=nv_mv_250")
soup = BeautifulSoup(page.text, "html.parser")
raw_html = soup.find("tbody", {"class": "lister-list"}).findAll("tr")

if __name__ == '__main__':
    v = []
    for html in raw_html:
        title = html.find("td", {"class":"titleColumn"}).find("a").get_text()
        year = html.find('span',{'class':'secondaryInfo'}).contents[0]
        rating = float(html.find("td", {"class":"ratingColumn imdbRating"}).find("strong").get_text())
        raw_list = [title, year, rating]
        v.append(raw_list)

    for x in range(len(v)):
        print(v[x][0], " ", v[x][1], " ", v[x][2])
user_113201420
Related