Sending GET request to URL with fragment returns the content of the main page

user_17835155 0
Viewed 50

I am trying to web-scrap this webpage but I always end up getting the "main" page (same URL but without "#face-a-face" at the end). It's the same problem as this guy encountered, see this forum. He got an answer but I am not able to generalize and apply this for the website I want to scrap.

import requests
from bs4 import BeautifulSoup

url_main = "https://www.lequipe.fr/Football/match-direct/ligue-1/2020-2021/ol-dijon-live/477168"
url_target = url_main + "#face-a-face"
soup_main = BeautifulSoup(requests.get(url_main, verify=False).content, "html.parser")
soup_target = BeautifulSoup(requests.get(url_target, verify=False).content, "html.parser")
print(soup_main == soup_target)

returns True. I would like to get different contents, this is not the case here.

For example, I would like to extract all the "confrontations depuis 2011" in the target webpage. How can I get the final content of this webpage with a GET request (or with another way) ? Thanks !

python-requests web-scraping beautifulsoup python-3.x python
1 Answers

All the data comes from a highly nested JSON file.

You can get that file and extract the information you need.

Here's how:

import json

import requests

endpoint = "https://iphdata.lequipe.fr/iPhoneDatas/EFR/STD/ALL/V2/Football/Prelive/68/477168.json"

team_data = requests.get(endpoint).json()
specifics = team_data["items"][1]["objet"]["matches"][0]["specifics"]

print(json.dumps(specifics, indent=2))

This should get you a dictionary:

{
  "__type": "specifics_sport_collectif",
  "vainqueur": "domicile",
  "score": {
    "__type": "score",
    "exterieur": "1",
    "domicile": "4"
  },
  "exterieur": {
    "__type": "effectif_sport_collectif",
    "equipe": {
      "__type": "equipe",
      "id": "202",
      "url_image": "https://medias.lequipe.fr/logo-football/202/{width}{declinaison}",
      "nom": "Dijon",
      "url_fiche": "https://www.lequipe.fr/Football/FootballFicheClub202.html"
    }
  },
  "domicile": {
    "__type": "effectif_sport_collectif",
    "equipe": {
      "__type": "equipe",
      "id": "22",
      "url_image": "https://medias.lequipe.fr/logo-football/22/{width}{declinaison}",
      "nom": "Lyon",
      "url_fiche": "https://www.lequipe.fr/Football/FootballFicheClub22.html"
    }
  },
  "is_final": false,
  "prolongation": false,
  "vainqueur_final": "domicile",
  "is_qualifier": false
}

And if you, for example, just want the socre, add this line:

just_the_score = specifics["score"]
print(just_the_score)

To get this:

{'__type': 'score', 'exterieur': '1', 'domicile': '4'}
user_61067910
Related