Beautifulsoup doesn't show all html elements

user_14494018 0
Viewed 161

I am fairly new to scraping and tried to get a list of companies from this webpage with this code:

import requests
import bs4
base_url = 'https://www.kimaventures.com/portfolio/'
res = requests.get(base_url)
soup = bs4.BeautifulSoup(res.text,"lxml")
soup

Does anyone know why I can't access all the html from the page?

Thanks for your help

web-scraping beautifulsoup python
2 Answers

You can do a get https://www.payitforward.vc/api/portfolio/VV7gXduVxznYhTZuZZP39Y/ to get at the JSON that the page loads. (Too busy to do this in python.)

curl "https://www.payitforward.vc/api/portfolio/VV7gXduVxznYhTZuZZP39Y/" ^
  -H "Connection: keep-alive" ^
  -H "Pragma: no-cache" ^
  -H "Cache-Control: no-cache" ^
  -H "Accept: application/json, text/plain, */*" ^
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" ^
  -H "DNT: 1" ^
  -H "Origin: https://www.kimaventures.com" ^
  -H "Sec-Fetch-Site: cross-site" ^
  -H "Sec-Fetch-Mode: cors" ^
  -H "Sec-Fetch-Dest: empty" ^
  -H "Referer: https://www.kimaventures.com/" ^
  -H "Accept-Language: en-US,en;q=0.9,fil;q=0.8"
Admin1

Based on Justin's answer:

import json
import requests


url = 'https://www.payitforward.vc/api/portfolio/VV7gXduVxznYhTZuZZP39Y/'
data = requests.get(url).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for d in data:
    print('{:<40} {:<30} {}'.format(d['name'], d['url'], d['description']))

Prints:

3D Shook                                 http://3dshook.com             The biggest 3D objects database tested and ready to be printed
77Foods                                  http://77foods.co              Making delicious bacon from plants
7 Elements Studios                       http://7elementsstudios.com    Creators of God of Axion
AB Tasty                                 http://abtasty.com             The best way to run A/B and multivariate tests for your app
Acces Telecom                            http://accesinternet.ca        We provide great Internet services in Canada's small cities
Accredible                               http://accredible.com          Course certificates for the information age
Acquire                                  http://acquire.app             Power Relevant  E-Commerce Marketing Experiences
ActiveEon                                http://activeeon.com           On-line cloud workflows to accelerate and scale applications
Adility                                  http://adility.com             Technology company enabling online to offline commerce transactions
Admedo - Adizio                          http://admedo.com              Self serve advertising platform across mobile and display
Adpushup                                 http://adpushup.com            Ad Revenue Optimization for Web Publishers
Adyoulike                                http://adyoulike.com           Native advertising technology, from content to distribution
Adyouneed                                http://adyouneed.com           Smartest way to create the ad you need.
Afrimarket                               http://afrimarket.fr           Innovative e-commerce platform in Africa
AfterSchool / ONE                        http://afterschoolapp.com      Anonymous and private message board for your school
Agentdesks                               http://agentdesks.com          A network of real estate agents powered by a mobile CRM. (AngelPad #8)

...and son on.
user_100359850
Related