How can I create a Python CSV file after scraping information from a web page?

user_14266865 0
Viewed 330

I am writing a code (yes, I am new with this) to extract information from a page on facebook. I am using facebook-scraper to get the info. I need to create a CSV file to store this information, but I always come out empty.

Original code

from facebook_scraper import get_posts
for post in get_posts('bibliotecaunespbauru', pages=66):
    print(post['time']) # não funciona
    print(post['post_id'])
    print(post['text'])
    print(post['image'])
    print(post['video'])
    print(post['likes'])
    print(post['comments'])
    print(post['shares'])
    print(post['link'])

Code to store in the CSV file.

import csv
from facebook_scraper import get_posts
for post in get_posts('bibliotecaunespbauru', pages=10):
    data = [print(post['post_id']), print(post['text']), print(post['image'])]
with open("facebook.csv", "w", newline="") as f:
   writer = csv.writer(f)
   writer.writerow(data)
with open('facebook.csv', newline='') as csvfile:
    data = csv.reader(csvfile, delimiter=' ')
    for row in data:
        print(', '.join(row))

Hey, many thanks for this. it makes so much sense now. However, it still doesn't work as now it retrieves only one request, instead of 10 pages.

import csv
from facebook_scraper import get_posts
for post in get_posts('bibliotecaunespbauru', pages=10):
     data = [post['post_id'], post['text'], post['image']]
with open("facebook.csv", "a", newline="") as f:
   writer = csv.writer(f)
   writer.writerow(data)
with open('facebook.csv', newline='') as csvfile:
    data = csv.reader(csvfile, delimiter=' ')
    for row in data:
        print(', '.join(row))

Third attempt. Stil getting just one post.


import csv
from facebook_scraper import get_posts
for post in get_posts('bibliotecaunespbauru', pages=10):
     data = [post['post_id'], post['text'], post['image']]
with open("facebook.csv", "a", newline="") as f:
   writer = csv.writer(f)
   writer.writerow(data)
with open('facebook.csv', newline='') as csvfile:
    data = csv.reader(csvfile, delimiter=' ')
    for row in data:
        print(', '.join(row))

Fourth attempt.

import csv
from facebook_scraper import get_posts
for post in get_posts('bibliotecaunespbauru', pages=10):
    data = [post['post_id'], post['text'], post['image']]
    with open("facebook.csv", "a", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(data)

Returns

UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-46-b4f7f9df1e02> in <module>
      5     with open("facebook.csv", "a", newline="") as f:
      6         writer = csv.writer(f)
----> 7         writer.writerow(data)

~\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py in encode(self, input, final)
     17 class IncrementalEncoder(codecs.IncrementalEncoder):
     18     def encode(self, input, final=False):
---> 19         return codecs.charmap_encode(input,self.errors,encoding_table)[0]
     20 
     21 class IncrementalDecoder(codecs.IncrementalDecoder):

UnicodeEncodeError: 'charmap' codec can't encode characters in position 76-77: character maps to <undefined>
facebook csv python
1 Answers

You have two issues with your code.

The first problem is with how you create data

Wrong

[print(post['post_id']), print(post['text']), print(post['image'])]

Why

In this line you are printing as you get the values, the return value of a print is None and hence None is stored in the list.

Old Output of data at each iteration: [None, None, None]

Correction:

[post['post_id'], post['text'], post['image']].

Output of new data: ['2092819824183367', 'Biblioteca da Unesp em Bauru ganha nova identidade visual ❤️\n\nhttps://youtu.be/dTCGp1eGmtM\n\nYOUTUBE.COM\nBiblioteca da Unesp em Bauru ganha nova identidade visual', None]

(PS : idk what that translates to)

Second problem being the way you write to a file.

open("facebook.csv", "w", newline="")

Note the a in open("facebook.csv", "a", newline="") while writing to a file, this is used to open a file for opening a file in "append" mode, opening it in w mode (your old code) will overwrite the file at each loop, causing a new blank file at every loop and this behavior is not what you need.

So integrating all changes and indentation, this is a complete code for what you expect

import csv
from facebook_scraper import get_posts
for post in get_posts('bibliotecaunespbauru', pages=10):
    data = [post['post_id'], post['text'], post['image']]
    with open("facebook.csv", "a", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(data)

Regarding the unicode errors

you can use open("facebook.csv", "a", newline="",encoding="utf-8") when you open the file

user_125029590
Related