How to save my links from BeautifulSoup in a text file with python?

user_16707482 0
Viewed 348

I'm learning python and webscraping, It is very cool but I am not able to get what I want. I'm trying to save products links in a text file to scrape data after. here is my script, which work correctly (almost) in the console of pycharm :

import bs4 as bs4
from bs4 import BeautifulSoup
import requests

suffixeUrl = '_puis_nblignes_est_200.html'
for i in range(15):
    url = 'https://www.topachat.com/pages/produits_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_page_est_' + str(i) + suffixeUrl
    response = requests.get(url)
    soup = bs4.BeautifulSoup(response.text, 'html.parser')
    if response.ok:
        print('Page: ' + str(i))

    for data in soup.find_all('div', class_='price'):
        for a in data.find_all('a'):
            link = (a.get('href'))
            links = ('https://www.topachat.com/' + link)
            print(links) #for getting link

My goal is to save the result of the links variable, line by line in a text file. I tried this, but something is wrong and I can't get each url :

            for link in links:
                with open("urls.txt", "a") as f:
                    f.write(links+"\n")

Please, does someone can help me?

web-scraping beautifulsoup python
3 Answers

You can do like this:

import bs4 as bs4
from bs4 import BeautifulSoup
import requests

suffixeUrl = '_puis_nblignes_est_200.html'
url_list = set()
for i in range(15):
    url = 'https://www.topachat.com/pages/produits_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_page_est_' + str(i) + suffixeUrl
    response = requests.get(url)
    soup = bs4.BeautifulSoup(response.text, 'html.parser')
    if response.ok:
        print('Page: ' + str(i))

    for data in soup.find_all('div', class_='price'):
        for a in data.find_all('a'):
            link = (a.get('href'))
            links = ('https://www.topachat.com/' + link)
            print(links) #for getting link
            url_list.add(links)


with open("urls.txt", "a") as f:
    for link in url_list:
        f.write(link+"\n")
user_95946130

Your problem is in for link in links line:

        link = (a.get('href'))
        links = ('https://www.topachat.com/' + link)
        print(links) 

        for link in links:
            with open("urls.txt", "a") as f:
                f.write(links+"\n")

Type of links is string and your for loop iterates it letter-by-letter (or characater-by-character). That is why you see a single character at each line in your txt file. You can just remove the for loop and the code will work:

from bs4 import BeautifulSoup
import requests

suffixeUrl = '_puis_nblignes_est_200.html'
for i in range(15):
    url = 'https://www.topachat.com/pages/produits_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_page_est_' + str(i) + suffixeUrl
    response = requests.get(url)
    soup = bs4.BeautifulSoup(response.text, 'html.parser')
    if response.ok:
        print('Page: ' + str(i))

    for data in soup.find_all('div', class_='price'):
        for a in data.find_all('a'):
            link = (a.get('href'))
            links = ('https://www.topachat.com/' + link)
            print(links) #for getting link
            with open("urls.txt", "a") as f:
                f.write(links+"\n")
user_111992980

You can try this way.

Just open the file once and write the complete data to it. Opening and closing files inside a loop is not a good thing to do.

import bs4 as bs4
from bs4 import BeautifulSoup
import requests

suffixeUrl = '_puis_nblignes_est_200.html'
with open('text.txt', 'w') as f:
    for i in range(15):
        url = 'https://www.topachat.com/pages/produits_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_page_est_' + str(i) + suffixeUrl
        response = requests.get(url)
        soup = bs4.BeautifulSoup(response.text, 'html.parser')
        if response.ok:
            print('Page: ' + str(i))

        for data in soup.find_all('div', class_='price'):
            for a in data.find_all('a'):
                link = 'https://www.topachat.com/' + a.get('href')
                f.write(link+'\n')

Sample output from text.txt

https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_ref_est_in11020650.html
https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_ref_est_in10119254.html
https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_ref_est_in20005046.html
https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_ref_est_in20002036.html
https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_ref_est_in20002591.html
https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_ref_est_in20004309.html
https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_ref_est_in20002592.html
https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_w_boi_sa_puis_ref_est_in10089390.html

.
.
.
user_27732060
Related