How to change platform when scraping a website (Futbin) in Python?

user_14082011 0
Viewed 129

I am currently looking at obtaining price data from Futbin specifically from this page of player data. I have used bs4 successfully for this with the following specific code:

spans = soup.find_all("span", class_="ps4_color font-weight-bold")

This collects all PS4 prices from the players page but I would like to also obtain Xbox and PC prices. To do this on the site you have to manually select it from the icons in the top right but from what I can tell this links to the same url but with updated price data. How can I scrape this data in a similar way to above as I'm sure there must be an easier way than using Selenium or similar packages.

Any help would be greatly appreciated!

python-requests web-scraping beautifulsoup python
1 Answers

To change page for another platform, set cookie= parameter in your request:

import requests
from bs4 import BeautifulSoup


url = 'https://www.futbin.com/20/players?page=1'
platforms = ['ps4', 'xone', 'pc']

for platform in platforms:
    print()
    print('Platform: {}'.format(platform))
    print('-' * 80)

    soup = BeautifulSoup( requests.get(url, cookies={'platform': platform}).content, 'html.parser' )

    for s in soup.select('span.font-weight-bold'):
        print('{:<40} {}'.format(s.find_previous('a', class_="player_name_players_table").text, s.text))

Prints:

Platform: ps4
--------------------------------------------------------------------------------
Lionel Messi                             2.5M 
Virgil van Dijk                          1.82M 
Cristiano Ronaldo                        3.2M 
Diego Maradona                           4.5M 
Pelé                                     6.65M 
Kevin De Bruyne                          1.95M 
Virgil van Dijk                          1.75M 
Lionel Messi                             2.18M 
Robert Lewandowski                       805K 
Cristiano Ronaldo                        3.08M 
Pelé                                     3.35M 
Kylian Mbappé                            2.62M 
Kevin De Bruyne                          1.21M 
Sadio Mané                               783K 
Kylian Mbappé                            2.66M 
Neymar Jr                                3.83M 
Diego Maradona                           2.19M 
Sadio Mané                               625K 
Alisson                                  148K 
N'Golo Kanté                             1.51M 
Robert Lewandowski                       269K 
Ronaldo                                  0 
Zinedine Zidane                          7.15M 
Lionel Messi                             4.6M 
Lionel Messi                             1.4M 
Alisson                                  143K 
Mohamed Salah                            459K 
Raphaël Varane                           847K 
Karim Benzema                            310K 
Luis Suárez                              407K 

Platform: xone
--------------------------------------------------------------------------------
Lionel Messi                             2.15M 
Virgil van Dijk                          1.65M 
Cristiano Ronaldo                        2.53M 
Diego Maradona                           4.07M 
Pelé                                     0 
Kevin De Bruyne                          1.73M 
Virgil van Dijk                          1.6M 
Lionel Messi                             1.9M 
Robert Lewandowski                       719K 
Cristiano Ronaldo                        2.51M 
Pelé                                     3.15M 
Kylian Mbappé                            2.27M 
Kevin De Bruyne                          1.02M 
Sadio Mané                               695K 
Kylian Mbappé                            2.24M 
Neymar Jr                                3.27M 
Diego Maradona                           1.61M 
Sadio Mané                               585K 
Alisson                                  153K 
N'Golo Kanté                             1.3M 
Robert Lewandowski                       247K 
Ronaldo                                  0 
Zinedine Zidane                          6.78M 
Lionel Messi                             4.26M 
Lionel Messi                             1.24M 
Alisson                                  130K 
Mohamed Salah                            470K 
Raphaël Varane                           725K 
Karim Benzema                            272K 
Luis Suárez                              351K 

Platform: pc
--------------------------------------------------------------------------------
Lionel Messi                             3.56M 
Virgil van Dijk                          2.5M 
Cristiano Ronaldo                        3.75M 
Diego Maradona                           4.3M 
Pelé                                     0 
Kevin De Bruyne                          2.52M 
Virgil van Dijk                          2.4M 
Lionel Messi                             2.86M 
Robert Lewandowski                       1.16M 
Cristiano Ronaldo                        3.75M 
Pelé                                     5.75M 
Kylian Mbappé                            3.35M 
Kevin De Bruyne                          1.4M 
Sadio Mané                               925K 
Kylian Mbappé                            3.3M 
Neymar Jr                                4.85M 
Diego Maradona                           1.98M 
Sadio Mané                               730K 
Alisson                                  179K 
N'Golo Kanté                             1.9M 
Robert Lewandowski                       400K 
Ronaldo                                  0 
Zinedine Zidane                          0 
Lionel Messi                             4.77M 
Lionel Messi                             2.3M 
Alisson                                  160K 
Mohamed Salah                            520K 
Raphaël Varane                           940K 
Karim Benzema                            370K 
Luis Suárez                              679K
user_100359850
Related