Databricks init scripts not working sometimes

user_4772836 0
Viewed 1638

Ok, it is very strange. I have some init scripts that I would like to run when a cluster starts

cluster has the init script , which is in a file (in dbfs)

basically this

dbfs:/databricks/init-scripts/custom-cert.sh

Now , when I make the init script like this, it works (no ssl errors for my endpoints. Also, the event logs for the cluster shows the duration as 1 second for the init script

dbutils.fs.put("/databricks/init-scripts/custom-cert.sh", """#!/bin/bash
cp /dbfs/orgcertificates/orgcerts.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
echo "export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt" >> /databricks/spark/conf/spark-env.sh
""")

However, if I just put the init script in an bash script and upload it to DBFS through a pipeline, the init script does not do anything. It executes , as per the event log but the execution duration is 0 sec.

I have the sh script in a file named

custom-cert.sh

with the same contents as above, i.e.

#!/bin/bash
cp /dbfs/orgcertificates/orgcerts.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
echo "export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt"

but when I check /usr/local/share/ca-certificates/ , it does not contain /dbfs/orgcertificates/orgcerts.crt, even though the cluster init script has run.

Also, I have compared the contents of the init script in both cases and it least to the naked eye, I can't figure out any difference

i.e.

%sh
cat /dbfs/databricks/init-scripts/custom-cert.sh

shows the same contents in both the scenarios. What is the problem with the 2nd case?

EDIT: I read a bit more about init scripts and found that the logs of init scripts are written here

%sh
ls /databricks/init_scripts/

Looking at the err file in that location, it seems there is an error

sudo: update-ca-certificates
: command not found

Why is it that update-ca-certificates found in the first case but not when I put the same script in a sh script and upload it to dbfs (instead of executing the dbutils.fs.put within a notebook) ?

EDIT 2: In response to the first answer. After running the command

dbutils.fs.put("/databricks/init-scripts/custom-cert.sh", """#!/bin/bash
cp /dbfs/orgcertificates/orgcerts.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
echo "export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt" >> /databricks/spark/conf/spark-env.sh
""")

the output is the file custom-cert.sh and then I restart the cluster with the init script location as dbfs:/databricks/init-scripts/custom-cert.sh and then it works. So, it is essentially the same content that the init script is reading (which is the generated sh script). Why can't it read it if I do not use dbfs put but just put the contents in bash file and upload it during the CI/CD process?

spark-bash-azure-databricks azure-databricks databricks
2 Answers

As we aware, An init script is a shell script that runs during startup of each cluster node before the Apache Spark driver or worker JVM start. case-2 When you run bash command by using of %sh magic command means you are trying to execute this command in Local driver node. So that workers nodes is not able to access . But based on case-1 , By using of %fs magic command you are trying run copy command (dbutils.fs.put )from root . So that along with driver node , other workers node also can access path .

Ref : https://docs.databricks.com/data/databricks-file-system.html#summary-table-and-diagram

user_95990910

It seems that my observations I made in the comments section of my question is the way to go.

I now create the init script using a databricks job that I run during the CI/CD pipeline from Azure DevOps.

The notebook has the commands

dbutils.fs.rm("/databricks/init-scripts/custom-cert.sh")
dbutils.fs.put("/databricks/init-scripts/custom-cert.sh", """#!/bin/bash
cp /dbfs/internal-certificates/certs.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
echo "export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt" >> /databricks/spark/conf/spark-env.sh
""")

I then create a Databricks job (pointing to this notebook), the cluster is a job cluster which is just temporary . Of course , in my case , even this job creation is automated using a powershell script.

I then call this Databricks job in the release pipeline using again a Powershell script.

This creates the file

/databricks/init-scripts/custom-cert.sh

I then use this file in any other cluster that accesses my org's endpoints (without certificate errors).

I do not know (or still understand), why can't the same script file be just part of a repo and uploaded during the release process (instead of it being this Databricks job calling a notebook). I would love to know the reason . The other answer on this question does not hold true as you can see, that the cluster script is created by a job cluster and then accessed from another cluster as part of its init script.

It simply boils down to how the init script gets created.

But I get my job done. Just if it helps someone get their job done too. I have raised a support case though to understand the reason.

user_47728360
Related