Response [400] when use file for parsing in python

user_10833715 0
Viewed 75

It is OK (response [200]) when I try to parse with manual texting but when I change the input from a file it becomes response [400].

This the code

import requests
from bs4 import BeautifulSoup

def people_spider():
    file = "D:\OneDrive\Documents\GPIP\Files\scraping\idtwitter.csv"
    dataset = open(file, "r")
    for account in dataset:
        href = 'https://twitter.com/' + account
        get_single_item_data(href)

def get_single_item_data(item_url):
    source_code = requests.get(item_url)
    print(source_code)
    print(item_url)
    plain_text = source_code.text
    soup = BeautifulSoup(plain_text, features='html.parser')
    for item_name in soup.findAll('p', {'dir': 'ltr'}):
        print(item_name.string)


people_spider()

and the result is

<Response [400]>
https://twitter.com/mr_adhani

<Response [400]>
https://twitter.com/RahayuNarti

<Response [400]>
https://twitter.com/AllMicroJobs

<Response [400]>
https://twitter.com/adibambang05

<Response [400]>
https://twitter.com/NatasyaRD1

<Response [400]>
https://twitter.com/arumyuniadis

<Response [400]>
https://twitter.com/harusan_osk

<Response [400]>
https://twitter.com/LailyFauziana

<Response [400]>
https://twitter.com/Dovia_Liata707

<Response [400]>
https://twitter.com/hapzah_putry

I have changed the extension too. However, it does not change any situation

web-scraping python
1 Answers

the problem is that you are not stripping account variable.

def people_spider():
    file = "D:\OneDrive\Documents\GPIP\Files\scraping\idtwitter.csv"
    dataset = open(file, "r")
    print(dataset)
    for account in dataset:
        href = 'https://twitter.com/' + account.strip()
        get_single_item_data(href)
user_107122740
Related