why dataflow is not supported by self hosted IR

Viewed 93

I know as per Azure documentation DataFlow is only allowed in AutoResolve IR (Azure IR), but looking for reason why is it not allowed in Self Hosted IR.!

1 Answers

Data flows requires a Spark environment, which the Azure IR provides. That serverless Spark environment is not provided by self-hosted IR. However, you can achieve the same things in Azure IR as you can in self-hosted IR in terms of accessing data in a vnet and on-prem.

Related