Different content when accessing a website with requests

user_16115803 0
Viewed 51

I am trying to get corresponding handle ids in ARIN automatically using a companies' name, like "Google".

https://search.arin.net/rdap/?query=google*

My naive approach is to use requests and BeautifulSoup:

import requests
from bs4 import BeautifulSoup

html = 'https://search.arin.net/rdap/?query='
comp = 'google*'

r = requests.get(html + comp)
soup = BeautifulSoup(r.text, 'html.parser')

#example search
search = soup.body.find_all(text = "Handle$")

However, I do not get the same output when I am using requests as when I simply use Google Chrome. The html code that is returned by requests is different and I cannot access the corresponding handles.

Does anyone know how to change the code?

python-requests beautifulsoup html python
1 Answers

The data you see on the page is loaded from external API URL. You can use requests module to simulate it:

import json
import requests

api_url = "https://rdap.arin.net/registry/entities"
params = {"fn": "google*"}

data = requests.get(api_url, params=params).json()

# pretty print the data:
print(json.dumps(data, indent=4))

Prints:

...

        {
            "handle": "GF-231",
            "vcardArray": [
                "vcard",
                [
                    [
                        "version",
                        {},
                        "text",
                        "4.0"
                    ],
                    [
                        "fn",
                        {},
                        "text",
                        "GOOGLE FIBER INC"
                    ],
                    [
                        "adr",
                        {
                            "label": "3425 MALONE DR\nCHAMBLEE\nGA\n30341\nUnited States"
                        },
                        "text",
                        [
                            "",
                            "",
                            "",
                            "",
                            "",
                            "",
                            ""
                        ]
                    ],
                    [
                        "kind",
                        {},
                        "text",
                        "org"
                    ]
                ]
            ],

...
user_100359850
Related