Many of my Azure ML Studio Designer pipelines began failing today. I was able to make a minimum repro:
Simply excluding columns with the Select Columns In Dataset node will fail with a JobConfigurationMaxSizeExceeded error.
This appears to be a bug introduced by Microsoft's rollout of their new Compute Common Runtime.
If I go into any nodes failing with the JobConfigurationMaxSizeExceeded exception and manually set AZUREML_COMPUTE_USE_COMMON_RUNTIME:false in their Environment JSON field, then they will subsequently work correctly. This is not documented anywhere that I could find, I stumbled over this fix through trial-and-error, and I wasted many hours trying to fix our failing pipelines today.
Does anyone know where I can find a list of possible effects of the Compute Common Runtime migration in Azure ML? I could not find any documentation on this and/or how it might affect existing Azure ML pipelines.