Scraping PornHub Video redirects to Cornhub

Question

I am making my first steps in webscraping and wanted to get Video Data from Pornhub.

In a first step i went trough all the pages on the main page and collected the video links. This worked and i got a csv with around 100k links. If i copy/paste those links to the brower , those work fine. BUT, when i go over them with my script to get my desired values, it always redirects me to a Cornhub Video (i know this was an april fools day joke some time ago). So it seems that my request gets redirected, but i dont know how this happens and if i can do anything about it.

'''with open("links.csv", "r") as f: lines = csv.reader(f)

for adress in lines:
   
    data = []
    print(data)
    headers = ({'User-Agent':
                'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'})


    sleep(randint(2,5))
   

    html = requests.get(adress[0], headers=headers)
    soup = BeautifulSoup(html.text, features="html.parser")

    print(soup)

   
    views = soup.find("script", type="application/ld+json")

    json_data = json.loads(views.contents[0])

    interaction_stat = json_data["interactionStatistic"]
    views = int(interaction_stat[0]
                ["userInteractionCount"].replace(",", ""))

    duration = int(
        soup.find("meta", property="video:duration").get("content"))

    upload_date = datetime.datetime.strptime(
        json_data["uploadDate"][0:10], '%Y-%m-%d').date()

    video_id = soup.find("form", id="shareToStream")
    video_id = video_id.find("input", id="attachment").get("value")

    data.append(video_id)
    data.append(upload_date)
    data.append(views)
    data.append(duration)

    with open("data.csv", "a", newline="") as f:  # Das hier über die schleife um es nur einmal zu machen
        writer = csv.writer(f) 
        writer.writerow(data)

'''

baduker · Answer

Your headers are really old, but this works just fine. Maybe make sure you alternate your IP or take some time before the subsequent requests.

import requests
from bs4 import BeautifulSoup
import json

lines = [
    "XX62f79e2ed1ed8",
    "XX63078405e84b6",
]

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Host": "www.pornhub.com",
    "Refer": "https://www.pornhub.com/",
}

with requests.Session() as s:
    for line in lines:
        soup = (
            BeautifulSoup(s.get(line, headers=headers).text, features="html.parser")
            .find("script", type="application/ld+json")
        )
        json_data = (
            json
            .loads(soup.getText())
            ['interactionStatistic'][0]['userInteractionCount']
        )
        print(json_data)

For the videos I've used the output is:

3,339,324
384,482

Scraping PornHub Video redirects to Cornhub

1 Answers