Using Jsoup to scrape Youtube

user_9930062 0
Viewed 147

I am trying to get some data from Youtube videos that the Youtube Data API doesn't provide. (Music Metadata when you press Read More)

Jsoup seems to be the right library to gather this data. I try to get the Document Object by calling

Jsoup.connect(videoUrl).get()

The problem now is that I always get the page where you get redirected to, to accept cookies. I also tried adding the userAgent but it didn't work.

Does anybody know how to get past the cookie page with Jsoup?
Or maybe any other idea to gather the music metadata

Thanks

jsoup youtube-data-api web-scraping youtube java
1 Answers

There are two possibilities:

  1. Click on the "accept" button for cookies with your web scraper (Selenium does it well)
  2. Manually click on "accept" and then refresh the webpage while checking the "Network" tab of your web browser with Ctrl+Shift+E. Copy as cURL with and without having clicked on the "accept" button you will find the difference and what parameter to provide to your web scraper to express your consent and bypass this menu.
user_71236600
Related