List all Blobs and containers of all the storage accounts

Question

I'm using the below script to run:

$context = Get-AzContext
$storageAccounts = Get-AzResource -ResourceType 'Microsoft.Storage/storageAccounts' 

[System.Collections.ArrayList]$saUsage = New-Object -TypeName System.Collections.ArrayList

 foreach ($storageAccount in $storageAccounts) {

   #list containers
   $containers= Get-AzStorageContainer -ResourceGroupName $storageAccount.ResourceGroupName -StorageAccountName $storageAccount.Name

     if($containers -ne $null){
          foreach($container in $containers){
            $StorageAccountDetails = [ordered]@{
                    SubscriptionName = $context.Subscription.Name
                    SubscrpitionID = $context.Subscription.Id
                    StorageAccountName = $storageAccount.Name
                    ContainerName = $container.Name
                    ResourceGroup = $storageAccount.ResourceGroupName
                    Location = $storageAccount.Location
                    size = $Storage.size
               }
             $saUsage.add((New-Object psobject -Property $StorageAccountDetails))  | Out-Null   
            }     
      }else{
      
        $StorageAccountDetails = [ordered]@{
                SubscriptionName = $context.Subscription.Name
                SubscrpitionID = $context.Subscription.Id
                StorageAccountName = $storageAccount.Name
                ContainerName = $null
                ResourceGroup = $storageAccount.ResourceGroupName
                Location = $storageAccount.Location
                size = $Storage.size      
         }
        $saUsage.add((New-Object psobject -Property $StorageAccountDetails)) | Out-Null
     }     
}
$saUsage

But I'm getting an error, Please assist

holger · Answer

The problem is described in the error message:

A parameter cannot be found that matches parameter name 'ResourceGroupName'

The cmdlet Get-AzStorageContainer does not support this parameter (and also does not suppport the -StorageAccountName parameter).

You might want to use the -Context parameter. However, this also requires you to use the Get-AzStorageAccount cmdlet since Get-AzResource does not return the Storage Account Context.

The context contains all relevant properties that are required to connect to the storage account.

Name               MemberType Definition
----               ---------- ----------
Equals             Method     bool Equals(System.Object obj)
GetHashCode        Method     int GetHashCode()
GetType            Method     type GetType()
ToString           Method     string ToString()
BlobEndPoint       Property   string BlobEndPoint {get;}
ConnectionString   Property   string ConnectionString {get;}
Context            Property   Microsoft.Azure.Commands.Common.Authentication.Abstractions.IStorageContext Context {get;set;}
EndPointSuffix     Property   string EndPointSuffix {get;}
ExtendedProperties Property   System.Collections.Generic.IDictionary[string,string] ExtendedProperties {get;}
FileEndPoint       Property   string FileEndPoint {get;}
Name               Property   string Name {get;set;}
QueueEndPoint      Property   string QueueEndPoint {get;}
StorageAccount     Property   Microsoft.Azure.Storage.CloudStorageAccount StorageAccount {get;}
StorageAccountName Property   string StorageAccountName {get;set;}
TableEndPoint      Property   string TableEndPoint {get;}

This is just for information. Using code along the lines below should solve this.

$storageAccounts = Get-AzStorageAccount

foreach ($storageAccount in $storageAccounts) {
    $containers = Get-AzStorageContainer -Context $storageAccount.Context
    # ... continue code
}

Please check the corresponding documentation for further details: https://docs.microsoft.com/en-us/powershell/module/az.storage/get-azstoragecontainer?view=azps-8.2.0

Karthikeyan Rasipalay Durairaj · Answer

You can use the below python version code.

from pyspark.sql.functions import col
from azure.storage.blob import BlockBlobService
from datetime import datetime
import os.path

account_name='container-name'
container_name ='folder1'
account_key = 'account-key'
prefix_val = 'data'

block_blob_service = BlockBlobService(account_name='%s'%(account_name), account_key='%s'%(account_key))

#block_blob_service.create_container(container_name)
generator = block_blob_service.list_blobs(container_name,prefix="%s"%(prefix_val),timeout=None)
#generator = block_blob_service.list_blobs(container_name)
report_time = datetime.now().strftime('%Y-%m-%d %H:%M:%S')


Target_file = "/dbfs/ADLSAUDIT/clouderadatalaketransfer/actian.csv" 
print(Target_file)

Target_file = open("%s"%(Target_file), 'w')

for blob in generator:
    
    length = BlockBlobService.get_blob_properties(block_blob_service,container_name,blob.name)
    last_modified = BlockBlobService.get_blob_properties(block_blob_service,container_name,blob.name).properties.last_modified
    file_size = BlockBlobService.get_blob_properties(block_blob_service,container_name,blob.name).properties.content_length
    blob_type = BlockBlobService.get_blob_properties(block_blob_service,container_name,blob.name).properties.blob_type
    if file_size != 0:
       line = account_name+'|'+container_name+'|'+blob.name+'|'+ str(file_size) +'|'+str(last_modified)[:10]+'|'
       print(line)
       Target_file.write(line+'
')
Target_file.close()

List all Blobs and containers of all the storage accounts

2 Answers