Can't replicate successful GET request from Postman in Google Apps Script

Question

I'm successfully making a GET request from Postman to this URI and it returns a valid response. The headers used are:

Host: asunnot.oikotie.fi
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0 Waterfox/56.3
Accept: application/json
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
OTA-token: 8552f1e070ca1c843bfdd22df45057d0111f1740411359b1209feea8c0b43b98
OTA-loaded: 1586629618
OTA-cuid: 5f97229e372238054461b228c11ae28de0f691b6
Referer: https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100
Cookie: AWSALB=TTLrvKn+28GOvkXt/3Mcen9O9n5kK68AngdNerJ312R8jh3zoc2XydYaah9p7Niu7rRWrIGys9lONCpG87rTTM9ba/OIK4jYhnRxqEYivrTq8Op+tG4oq9B0F4Il; AWSALBCORS=TTLrvKn+28GOvkXt/3Mcen9O9n5kK68AngdNerJ312R8jh3zoc2XydYaah9p7Niu7rRWrIGys9lONCpG87rTTM9ba/OIK4jYhnRxqEYivrTq8Op+tG4oq9B0F4Il; cardType=100; instance=9a1337a3128a8b73ae9beeff3fe2c1a7; PHPSESSID=a4112077ed3095154257d8b1759c8af5; user_id=5f97229e372238054461b228c11ae28de0f691b6
Connection: keep-alive
If-Modified-Since: Sat, 11 Apr 2020 18:25:00 GMT
Cache-Control: max-age=0

Now, when I try to replicate the same request with google apps script, I get a 401 unauthorized request (to the same URI with the same header). What gives?

This is the code used (using encodeURI because otherwise I get an 'Exception: Invalid argument'):

function testGetJSON(){ 
  var url = 'https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType[]=1&conditionType[]=2&limit=24&locations=[[1669,4,"Lauttasaari,+Helsinki"],[14714,5,"00340,+Helsinki"]]&lotOwnershipType[]=1&offset=0&price[max]=600000&price[min]=150000&roomCount[]=3&size[min]=35&sortBy=published_sort_desc';
  var res = encodeURI(url);
  var opt = {
            "method": "GET", 
            "muteHttpExceptions": true,
            // "escaping":false,
      headers: {
        "Host": "asunnot.oikotie.fi",
        "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0 Waterfox/56.3",
        "Accept": "application/json",
        "Accept-Language": "en-US,en;q=0.5",
        "Accept-Encoding": "gzip, deflate, br",
        "OTA-token": "8552f1e070ca1c843bfdd22df45057d0111f1740411359b1209feea8c0b43b98",
        "OTA-loaded": "1586629618",
        "OTA-cuid": "5f97229e372238054461b228c11ae28de0f691b6",
        "Referer": "https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100",
        "Cookie": "AWSALB=TTLrvKn+28GOvkXt/3Mcen9O9n5kK68AngdNerJ312R8jh3zoc2XydYaah9p7Niu7rRWrIGys9lONCpG87rTTM9ba/OIK4jYhnRxqEYivrTq8Op+tG4oq9B0F4Il; AWSALBCORS=TTLrvKn+28GOvkXt/3Mcen9O9n5kK68AngdNerJ312R8jh3zoc2XydYaah9p7Niu7rRWrIGys9lONCpG87rTTM9ba/OIK4jYhnRxqEYivrTq8Op+tG4oq9B0F4Il; cardType=100; instance=9a1337a3128a8b73ae9beeff3fe2c1a7; PHPSESSID=a4112077ed3095154257d8b1759c8af5; user_id=5f97229e372238054461b228c11ae28de0f691b6",
        "Connection": "keep-alive",
        "If-Modified-Since": "Sat, 11 Apr 2020 18:25:00 GMT",
        "Cache-Control": "max-age=0",
}
};

  var str = UrlFetchApp.fetch(res,opt);
  Logger.log(str.getContentText());
}

Where am I going wrong?

Bonus: What is the easiest way to turn the block of header parameters above (colon separated, without quotes or commas) to the one needed in GAS? (without manually adding all the quotes and commas)

user_1202124 · Answer

A way to get around this issue is to simply use the option 'copy as... > fetch' within the browser inspector (in my case Opera), which gives out the following (somewhat different headers than the ones sent by the browser):

fetch("https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,+Helsinki%22%5D,%5B14714,5,%2200340,+Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc", {"credentials":"omit","headers":{"accept":"application/json","ota-cuid":"fd2a3a03d52a2721f9a9aa844ddf7eef2ac66ed6","ota-loaded":"1586685082","ota-token":"ab7e9f830a7dff3a9b01fbdcbc899ed7bfa659a4793103f1943e83ef5f938b16","sec-fetch-dest":"empty"},"referrer":"https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100","referrerPolicy":"no-referrer-when-downgrade","body":null,"method":"GET","mode":"cors"});

Which I used in GAS in the following way:

function testGetJSON(){
 var str = UrlFetchApp.fetch("https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,+Helsinki%22%5D,%5B14714,5,%2200340,+Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc",
 {"credentials":"omit",
"headers":{"accept":"application/json","ota-cuid":"fd2a3a03d52a2721f9a9aa844ddf7eef2ac66ed6","ota-loaded":"1586685082","ota-token":"ab7e9f830a7dff3a9b01fbdcbc899ed7bfa659a4793103f1943e83ef5f938b16","sec-fetch-dest":"empty"},"referrer":"https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100","referrerPolicy":"no-referrer-when-downgrade","body":null,"method":"GET","mode":"cors"});
  Logger.log(str.getContentText());
}

Can't replicate successful GET request from Postman in Google Apps Script

1 Answers