azure experts
I am a DS recently trying to manage the pipelines after our DE left, so quite new to azure. I recently ran into an issue that our azure storage usage increased quite significantly, I found some clues, but now stuck and will be really grateful if an expert can help me through.
- Looking into the usage, it seems the usage increase to be mainly due to a significant increase of transaction and ingress of the API ListFilesystemDir. I traced the usage profile back to the time when our pipelines run, I located the main usage to be from a copy data activity, which copies data from a folder in ADLS to SQL DW staging table. The amount of data being actually transferred is only tens of MB, similar to before, so should not be the reason leading to the substantial increase. The main difference I found is in the transaction and ingress using the API ListFilesystemDir, which shows an ingress volume of hundreds of MB and ~500k transactions during the pipeline run. The surprising thing is, the usage of this API ListFilesystemDir was very small before (at tens of kB and ~1k transaction). recent storage usage of API ListFilesystemDir
- Looking at the pipeline, I notice it uses a filepath wildcard to filter and select all the most recently updated files within a higher-level folder. pipeline
My questions are
- Is there any change to the usage/limit/definition of API ListFilesystemDir that could cause such an issue?
- Does the application of wildcard path filter means azure need to query all the file lists in the root folder and then do filter, such it leads to a huge amount the transaction? Although it didn't seem to cause the issue before, has there been any change to the API usage of this wildcard filter?
- Any suggestion on solving the problem?
Thank you very much!