Curl - different result in node (child process) and terminal

Question

So I accidentally found something weird and can't figure out why is it.

I make this curl in my terminal (which is ZSH on iTerm):

curl -I https://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer/find\?text\=1%20TX%2077979\&f\=json

And the response is:

HTTP/2 200
date: Wed, 28 Oct 2020 17:31:12 GMT
content-type: application/json;charset=UTF-8
content-length: 374
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
server:
etag: dc0a3f17
x-esri-ftiles-cache-compress: true
cache-control: max-age=300
x-cached: MISS
vary: Origin,Accept-Encoding
strict-transport-security: max-age=31536000

But if I run the same curl as exec child_process in node (v14.14.0) like this:

const { exec } = require('child_process');
const url = 'https://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer/find\?text\=1%20TX%2077979\&f\=json';

return exec(`curl -I ${url}`, (err, std) => {
  console.log(std);
});

The response look like this:

HTTP/2 403 
date: Wed, 28 Oct 2020 17:17:18 GMT
content-type: text/html;charset=utf-8
content-length: 680
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
server:

Can anyone explain it to me?

----- EDIT -----

To andrew: I was using urls that contains '&' but they ended with '&query=' which is not obligatory and it still returned HTTP 200 and other needed info, that's why I didn't even notice.

user_10588408 · Answer

As @drew010 wrote:

As you found, you need to quote the URL if it contains & character since this will be interpreted by the shell as a control operator. If & appears anywhere in your command or arguments and is not quoted or escaped & I think you'd run into the same issue all the time.

Ok, so according to a documentation (https://nodejs.org/api/child_process.html#child_process_child_process_exec_command_options_callback)

The command string passed to the exec function is processed directly by the shell and special characters (vary based on shell) need to be dealt with accordingly:

So after performing 'echo' the same way I found out that end of url ("&f=json") is being removed.

After adding quotes around url variable interpolation it works fine.

exec(`curl -I "${url}"`, (err, std) =>

My initial confusion "why is it different when it should do the same" was also part of my believe that 'exec' would just pass it to the terminal I'm using, which as I mentioned before is iTerm with ZSH, because it is set as default terminal on my OS. That assumption was wrong and it is mentioned in the documentation:

shell Shell to execute the command with. See Shell requirements and Default Windows shell. Default: '/bin/sh' on Unix

So it is not using default OS terminal, but just sh. But it is possible to use whichever you want with 'shell' option.

Curl - different result in node (child process) and terminal

1 Answers