How to get the link set by javascript with puppeteer

user_11734835 0
Viewed 133

At https://www.popcornflix.com/channels/details/actionthrillers , I'm trying to get the link that the browser goes to when you click on a movie. The selector for the first movie is:

#main-view > div > ui-view > div > div > div:nth-child(2) > div > div.ng-scope > div > div:nth-child(1)

but there doesn't seem to be an href anywhere inside that div so I'm assuming each time you click on it it will be set by javascript with the attribute 'ng-click'. This is the element itself:

<div class="grid-item-container grid-video-container thumbrow-item ng-scope" ng-repeat="item in vm.items" ng-include="&quot;includes/movie-card-item.html&quot;" ng-click="vm.onItemClick($event, $index, item)"></div>

How do I go about getting the URL without having to click on it first? Or is there a way to force it to open the link in another tab? I'm trying to not leave the current tab. Thanks in advance

puppeteer html javascript
1 Answers

As you already suspecting: it would be only possible to collect the links with browser automation if you'd click the links then save the current url.

You could only force open the links on new tabs if you'd use browser.newPage and page.goto multiple times. But it would require to know the list of links, what you don't have yet.

In theory

If you would want to retrieve the links you could study the URLs of the movie pages, how the URL-s structured, then you'd be able to concat them in the baseurl + path + slug format. E.g.:

const url = 'https://www.popcornflix.com/' + 'movie/' + '18-v0b12bzgha2f-mutant-chronicles'

(URL parameters are not mandatory to access a page)

Every single page apps are feed from an api endpoint which is mostly avaialble in the browser's network, so in this case the ng-click is performed based on JSON data coming from the api. With minimal reverse engineering effort you could retrieve the page's slug from the JSON.

const apiURLpattern = '...'
let apiResponse
page.on('response', async response => {
  if (response.url().includes(apiURLpattern)) {
    apiResponse = await response.json()
    }
  })
await page.goto(...

... in theory.

Unfortunately it is not allowed by the owner of the site: https://www.popcornflix.com/pages/about/a/terms

15. Prohibited Uses of the Site.

(o) You agree not to reverse engineer, decompile, disassemble or otherwise attempt to discover the source code of the Site or any part thereof, except and only to the extent that such activity is expressly permitted by applicable law notwithstanding this limitation.

The same paragraph of terms doesn't allow scraping neither:

(j) You agree that you will not use any robot, spider, scraper, or other automated means to access the Site for any purpose without our express prior written permission or bypass our robot exclusion headers or other measures we may use to prevent or restrict access to the Site.

Suggestion

I suggest to contact the owners and ask access to their API if you need it for personal purposes. Or if you need movie details: TMDb's api is free to use and has huge database: https://developers.themoviedb.org/3

user_124125950
Related