What is the Flow Run Size Limit of a notebook activity in Azure Synapse?

user_13568627 0
Viewed 428

I created a large spark notebook and ran it successfully in Azure Synapse. Then I created a new pipeline with a new notebook activity pointing to the existing spark notebook. I triggered it and it failed with the error message:

ErrorCode=FlowRunSizeLimitExceeded, ErrorMessage=Triggering the pipeline failed 
due to large run size. This could happen when a run has a large number of 
activities or large inputs used in some of the activities, including parameters.

There is only one activity in that pipeline; so, it can't be the number of activities being exceeded. I googled flow run size limit on activity and there was no result. What is the flow run size limit on the notebook activity?

Here is the information:

Filename Blob Size
UID_ISO_FIPS_LookUp_Table.csv 396 KiB
05-11-2021.csv 630 KiB
https://ghoapi.azureedge.net/api 476 KiB
Type Size Cell Total Cluster Size
.ipynb notebook 668,522 bytes 43 cells Small (4 vCores / 32 GB) - 3 to 3 nodes

Here is the error message after triggering the pipeline error message after triggering the pipeline

Here is the sample code in the notebook. The purpose is to join three files into a single file with a single table. Some processing of csv files are filtering, selecting columns, renaming columns, and aggregating values.

enter image description here

sample code

Could someone explain why the error message occurred?

azure-synapse
1 Answers

I was able to import that .csv with the following Python code on a small Spark pool:

%%pyspark
df = spark.read.load('abfss://someContainer@somestorageAccount.dfs.core.windows.net/raw/csv/05-11-2021.csv', format='csv'
, header = True
)
display(df.limit(10))

df.createOrReplaceTempView("tmp")

Saving it as a temp view allows you to write some conventional SQL to query the dataframe, eg

%%sql
SELECT SUM(deaths) xsum
FROM tmp
user_15275040
Related