how to grab next marker (next_marker) in azure cli command az storage fs file list

user_665550 0
Viewed 1336

Need to calculate size of specific containers and folders at ADLS Gen2. Started with command az storage fs file list. However don't understand how to grab next_marker ? It appears in stdout as warning but not in output of command:

WARNING: Next Marker:
WARNING: VBbYvMrBhcCCqHEYSAAAA=

So how to get this next_marker:

$files=$(az storage fs file list --file-system <container name>\
 --auth-mode login --account-name <account name> \
 --query "[*].[contentLength]" --num-results 1000 -o tsv)

$files.next_marker is empty.

UPDATE1: Created issues https://github.com/Azure/azure-cli/issues/16893

azure-data-lake-gen2 azure-cli2 azure-cli azure-powershell azure
1 Answers

If you're using this azure cli command: az storage fs file list, the next_marker is not returned to the variable $files, it's always printed out in the console. You need to copy and paste it.

As a workaround, you can use this azure cli command: az storage blob list(Most of the azure blob storage cli commands are also available in ADLS Gen2). This command has a parameter --show-next-marker, you can use it to return next_marker to a variable.

I write an azure cli scripts and it can work well for ADLS Gen2:

 $next_token = ""
 $blobs=""
 $response = & az storage blob list --container-name your_file_system_in_ADLS_Gen2 --account-name your_ADLS_Gen2_account --account-key your_ADLS_Gen2_key --num-results 5 --show-next-marker | ConvertFrom-Json
 $blobs += $response.properties.contentLength
 $next_token = $response.nextMarker 

  while ($next_token -ne $null){
    
  $response = & az storage blob list --container-name your_file_system_in_ADLS_Gen2 --account-name your_ADLS_Gen2_account --account-key your_ADLS_Gen2_key --num-results 5 --marker $next_token --show-next-marker | ConvertFrom-Json
  
  $blobs = $blobs + " " + $response.properties.contentLength

  $next_token = $response.nextMarker 
  }

$blobs

The test result:

enter image description here

Please note that upgrade your azure cli to the latest version, the --show-next-marker parameter may not work in the old versions as per this issue.

user_101858160
Related