We have two different Azure cloud resource groups, RG1 and RG2, where RG1 hosts the ADB_source of the data source, and RG2 hosts the ADB_sink & ADLS_sink(gen2) of the data sink.
Use Case: We have a few delta tables in ADB_source (ACL enabled) where a list of users has Read access. In the ADB_source workspace, we need to read the delta tables and write them into ADLS_sink as parquet for further processing at the sink.
What's Available: We have a high concurrency cluster created in ADB_Source workspace, which -
- Allows only Python & SQL (dbutils.fs also restricted).
- Credential Passthrough is disabled.
- Has ACLs Enabled in spark config.
- Has mount point created to a container in ADLS_sink.
- Has no Admin Access to the cluster.
Errors Observed:
We could read the delta tables as expected and run action commands as long as they are in the ADB_source workspace. However, when we write that data into the ADLS_sink with .save(), we get the below error.
Py4JJavaError: An error occurred while calling o410.save. : java.lang.SecurityException: User does not have permission SELECT on any file. User does not have permission MODIFY on any file.
I would appreciate it if anyone could explain this and recommend additional security checks/accesses needed to implement the use case successfully.