I need help generating the correct authorization header to POST to Azure Tables using curl on Linux. I'm coming in cold to the world of API and REST. I've been using the az cli to do my merges but under load it chews up all resource on the box so the ultimate aim is to script batch inserts from within a ksh script. For now I'm just trying to get an insert working, but if I'm on the wrong track any insights on the batch goal would be very welcome.
This question Azure Storage Table API REST with Curl has helped a lot because it is the only complete example I have found that does something very similar from a nix command line (rather than all the C# examples MS provide). Using this I have been able to query (GET) full tables, specific entities/values etc without curl -k (insecure) - all good. Transforming this into an insert is proving problematic.
I've borrowed almost everything from the above question. The only bits I've changed are GET>POST, accept>Content-Type, dropped the Content-Length and added the cacert file I downloaded from the suggested curl website. This is the curl attempt...
# curl -v \
> -H "Authorization: SharedKey XXXXXXX:${signature}" \
> -H "x-ms-date: $request_date" -H "x-ms-version: 2017-07-29" \
> -H "Content-Type: application/json; charset=UTF-8" \
> --data @body.json "https://XXXXXXX.table.core.windows.net/YYYYYYY" \
> --cacert cacert.pem
* Trying <20.150.40.102>...
* TCP_NODELAY set
* Connected to XXXXXXX.table.core.windows.net (20.150.40.102) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: cacert.pem
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=*.table.core.windows.net
* start date: May 19 11:38:54 2020 GMT
* expire date: May 19 11:38:54 2022 GMT
* subjectAltName: host "XXXXXXX.table.core.windows.net" matched cert's "*.table.core.windows.net"
* issuer: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation; OU=Microsoft IT; CN=Microsoft IT TLS CA 4
* SSL certificate verify ok.
> POST /YYYYYYY HTTP/1.1
> Host: XXXXXXX.table.core.windows.net
> User-Agent: curl/7.62.0-DEV
> Accept: */*
> Authorization: SharedKey XXXXXXX:UJs5ATaZZZZZZZZ1L3c=
> x-ms-date: Fri, 29 May 2020 13:58:52 GMT
> x-ms-version: 2017-07-29
> Content-Type: application/json; charset=UTF-8
> Content-Length: 183
>
* upload completely sent off: 183 out of 183 bytes
< HTTP/1.1 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
< Content-Length: 419
< Content-Type: application/xml
< Server: Microsoft-HTTPAPI/2.0
< x-ms-request-id: ccf4784c-4002-000a-44c1-3556f6000000
< x-ms-error-code: AuthenticationFailed
< Date: Fri, 29 May 2020 13:58:53 GMT
<
<?xml version="1.0" encoding="utf-8"?><m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><m:code>AuthenticationFailed</m:code><m:message xml:lang="en-US">Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:ccf4784c-4002-000a-44c1-3556f6000000
* Connection #0 to host XXXXXXX.table.core.windows.net left intact
Time:2020-05-29T13:58:53.9256087Z</m:message></m:error>
That second ALPN message appears to be a non issue (according to google) and that handshake looks good (right?). So, I think I'm good in terms of certificates, keys etc. I must have messed up the format of the string_to_sign.
The required headers vary between verbs. Referring to the MS docs, they advise the following required headers and say that order is important...
StringToSign = VERB + "\n" +
Content-MD5 + "\n" +
Content-Type + "\n" +
Date + "\n" +
CanonicalizedResource;
Only verb, date and canon are required, so...
POST\n\n\n${request_date}\n/XXXXXXX/YYYYYYY;
...should be fine. However, there is also a footnote that says I need 'DataServiceVersion and MaxDataServiceVersion headers' but it's not included in the example directly above it and there's no indication of the order required for these additional fields. https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#Subheading2.
Googling further it seems that these actually need to be in the json, so I put them into my body.json file.
{
"M00":"98989898",
"PartitionKey":"20200529",
"RowKey":"sometext"
"DataServiceVersion":"3.0;NetFx"
"MaxDataServiceVersion":"3.0;NetFx"
}
Extra Credit... Looking further ahead to the batch configuration I can see that there's more 'fun' on the way so if anyone has done this from bash/ksh/nix command line please do share. I sense I'm getting close but I'm finding it difficult to envisage what I need to construct on the command line when I look at this for example...
https://docs.microsoft.com/en-us/rest/api/storageservices/performing-entity-group-transactions
(And where do those batch and changeset IDs come from??)