How to upgrade python version in Databricks

user_1232087 0
Viewed 302

We upgraded Databricks from 10.3 to 10.4 LTS. But the python version did not change from python 3.8.10.

Question: In Databricks - version 10.4, how can we upgrade the python version from python 3.8.10 to python 3.10?

UPDATE: I would like to use some new functionalities offered in python 3.10 such as match case Statement.

azure-databricks databricks python-3.x python
2 Answers

You might be able to install python 3.10.5 on a Docker image that a cluster can utilise instead of the standard runtime.

https://docs.databricks.com/clusters/custom-containers.html

You can build upon the minimal configuration. I have made a minimal example

FROM databricksruntime/minimal:experimental

# Installs python 3.10 and virtualenv for Spark and Notebooks
RUN apt-get update \
  && apt-get install -y \
    python3.10 \
    virtualenv \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Initialize the default environment that Spark and notebooks will use
RUN virtualenv -p python3.10 --system-site-packages /databricks/python3

# Specifies where Spark will look for the python process
ENV PYSPARK_PYTHON=/databricks/python3/bin/python3

You will need to install all other python libraries, so the process is a bit more tedious.

user_196999490

It might not be possible to upgrade the version of python inside a Databricks cluster. Each cluster have a pre-defined configuration which consist of specific versions of Spark, Scala and Python.

We upgraded Databricks from 10.3 to 10.4 LTS. But the python version did not change from python 3.8.10

  • This is because both Databricks 10.3 and 10.4 LTS have python version as 3.8.10.

One solution would have been to Edit the cluster to change to Databricks runtime which supports required configuration. To do this Navigate to compute -> click on your cluster -> edit and choose required databricks runtime.

But currently, the highest python version supported in Azure databricks is Python 3.9.5 by Databricks runtime 11.1. Refer to this Microsoft documentation to understand more about features and configurations of Databricks runtimes

user_188445850
Related