Node `https.get` returns 403, `axios.get` and `curl` return 200

user_6951428 0
Viewed 721

I'm trying to perform a GET request using Node's native https module and a 3rd party axios module (which is pretty commonly used):

const endpoint = 'https://www.pathofexile.com/api/trade/data/stats';

require('axios')
    .get(endpoint)
    .then(response => console.log('axios', response.status));
// prints 200

require('https')
    .get(endpoint, {},
        response => console.log('https', response.statusCode));
// prints 403

The axios request works fine as expected; I can view response.data as well. But the https request fails with a 403.

  • Strangely, both used to work last time I checked a few months ago.
  • I haven't updated Node on my machine (node v 14.5.0). I've also tried on another machine as well (13.3.0).
  • curl from the command line also works fine.

Since https used to work, my hunch is that the server must have changed something, and that https.get and axios.get must be doing something slightly differently (e.g. sending different headers by default) that the server is no longer indifferent to.

axios node.js javascript
2 Answers

After a little bit of testing, I noticed that by default the https module does not include an User-Agent header, while axios and curl both do. The server is rejecting your request because you have no User-Agent set.

To solve this, simply specify a custom User-Agent of your choice:

require('https')
    .get(endpoint, {headers: {'User-Agent': 'whatever'}},
        response => console.log('https', response.statusCode));
user_38894490

After few web results focusing on node https didn't find an answer, I found another stackoverflow post about python gets returning 403 while curl returns 200.

It turns out it was an issue with the request's 'User-Agent' header.

Curl by default sends the user agent 'curl/7.72.0':

> GET /94548/2401383/manifest.js HTTP/1.1
> Host: cdn.flashtalking.com
> User-Agent: curl/7.72.0
> Accept: */*

Axios by default sends the user agent 'axios/0.20.0\r\n':

_header: 'GET /api/trade/data/stats HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'User-Agent: axios/0.20.0\r\n' +
      'Host: www.pathofexile.com\r\n' +
      'Connection: close\r\n' +
      '\r\n',

While node's https by default sends no user agent:

  _header: 'GET /api/trade/data/stats HTTP/1.1\r\n' +
    'Host: www.pathofexile.com\r\n' +
    'Connection: close\r\n' +
    '\r\n',

Apparently, the server now checks for a non empty User-Aagent header. Updating the https request to do so gets a 200:

const endpoint = 'https://www.pathofexile.com/api/trade/data/stats';

require('axios')
    .get(endpoint)
    .then(response => console.log('axios', response.status));
// prints 200

require('https')
    .get(endpoint, {},
        response => console.log('https', response.statusCode));
// prints 403


require('https')
    .get(endpoint, {header: {headers: {'User-Agent': '_'}}}},
        response => console.log('https', response.statusCode));
// prints 200
user_69514280
Related