Is there an efficient way of scraping and AdfPage?

Question

I'm trying to scrape an ADF Page, but I'm stuck in the loop if I don't use an actual browser.

I'm currently using puppeteer to find the two parameter values I need:

const sessionParams = async () => {
  const browser: Browser = await puppeteer.launch()
  const page: Page = await browser.newPage()
  await page.goto(
    'https://sia.unal.edu.co/ServiciosApp/facespublico/public/servicioPublico.jsf?taskflowId=task-flow-AC_CatalogoAsignaturas',
    { waitUntil: 'domcontentloaded' }
  )

  const params = {
    windowId: await page.$eval('input[name="Adf-Window-Id"]', (input) => {
      return input.getAttribute('value')
    }),
    viewState: await page.$eval('input[name="javax.faces.ViewState"]', (input) => {
      return input.getAttribute('value')
    }),
  }

  await browser.close()
  return params
}

This works, but it is painfully slow (around 15 seconds). Using fetch or similar libraries does not work, since the value of these inputs will read: winnoloop instead of the random generated string.

It seems like the page was made using ADF Faces.

What I eventually want is to be able to send this request:

curl "https://sia.unal.edu.co/ServiciosApp/facespublico/public/servicioPublico.jsf?Adf-Window-Id=w18k7uelfon&Adf-Page-Id=0" -X POST -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0" -H "Accept: */*" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" -H "Adf-Ads-Page-Id: 1" -H "Adf-Rich-Message: true" -H "Origin: https://sia.unal.edu.co" -H "DNT: 1" -H "Connection: keep-alive" -H "Referer: https://sia.unal.edu.co/ServiciosApp/facespublico/public/servicioPublico.jsf?taskflowId=task-flow-AC_CatalogoAsignaturas" -H "Cookie: ALUMNO_ID=portal2|YVgHF|YVfeG; OAMAuthnCookie_sia.unal.edu.co:443=QnD9HduBUygNYeC8ji173A6UmrNG1UET3DXA5"%"2FedKuQHJXHZqLafbnG37lRmumYlggWxbC"%"2BWE39peBn54rsxgOsMqBVG2pC9"%"2BFxof0c"%"2BzUFBJTaCBLrlTvcl2"%"2BC97CrkuvSFf94ChwtJIXZKgG5tcUe9OYgO6aj11INP8ccb5auLWJQE3nj1wQp0nrgKUC"%"2BH87M1TiTEU7Rtoi3hmKD82p3"%"2FXPtxEmjSTPzzB3M47ExJNsIc6"%"2FhKfpGnGVIpDQmnq9T5T5c"%"2Brptm"%"2FFjaPCE2W8WIDXeI4Y2ECaymlTqBK5p67VZ4edD333QkaGdmIedBEi4kFRzlnqQ7EennU7oSCu3"%"2BJFnRg87EtLiZXz0rcLaKi1"%"2B3arJgcdBcS"%"2FwCwGRCeTR9IhMnWE9Uv"%"2F9mrPreQYwES6KpLWtJMh5FI9JjXkykOr4wrSqsR2HZWiuBOqnY0KOgkJpNYnvUpONN15TmiOthyx"%"2FRNMrdsW5CLfOh4lzpFnhhmh4izwg"%"2BpvLqyNU"%"2BVHalSob0oMUkgD7W3Jpa4xzTM03r7ggtO"%"2FOFUXeCC"%"2Fg"%"3D; PortalJSESSION=pqI_22pLl_RHReoaBuDzV-5NpujqeyD53-5h_kKioezZJAgw0t1w!-1695300558" -H "Sec-Fetch-Dest: empty" -H "Sec-Fetch-Mode: cors" -H "Sec-Fetch-Site: same-origin" --data-raw "pt1:r1:0:soc1=0&pt1:r1:0:soc9=6&pt1:r1:0:soc2=4&pt1:r1:0:soc3=12&pt1:r1:0:soc4=&pt1:r1:0:soc5=&pt1:r1:0:soc10=0&pt1:r1:0:it10=&pt1:r1:0:it11=&org.apache.myfaces.trinidad.faces.FORM=f1&Adf-Window-Id=w18k7uelfon&Adf-Page-Id=0&javax.faces.ViewState=!-um1yuq943&event=pt1"%"3Ar1"%"3A0"%"3Acb1&event.pt1:r1:0:cb1="%"3Cm+xmlns"%"3D"%"22http"%"3A"%"2F"%"2Foracle.com"%"2FrichClient"%"2Fcomm"%"22"%"3E"%"3Ck+v"%"3D"%"22type"%"22"%"3E"%"3Cs"%"3Eaction"%"3C"%"2Fs"%"3E"%"3C"%"2Fk"%"3E"%"3C"%"2Fm"%"3E&oracle.adf.view.rich.PROCESS=pt1"%"3Ar1"%"2Cpt1"%"3Ar1"%"3A0"%"3Acb1"

But since the Adf-Window-Id and the javax.faces.ViewState "expire", I cannot use the same ones for all requests.

Is there an efficient way of scraping and AdfPage?

0 Answers