Trying to install H2O on Databricks. How can I install a 3rd party package in Databricks?

user_5212614 0
Viewed 583

I'm playing around with Databricks now, and looking at a demo online. The demo requires H2O to be installed. In Databricks, I tried the following:

   click clusters (then click on the name of the cluster)
   click libraries
   click install new
   finally, click maven

In the coordinates box, I paste the following:

$SPARK_HOME/bin/spark-shell --packages ai.h2o:sparkling-water-package_2.12:3.30.0.6-1-3.0

I am following the instructions from the links below.

https://docs.databricks.com/applications/machine-learning/third-party/index.html

http://docs.h2o.ai/sparkling-water/3.0/latest-stable/doc/install/install_and_start.html#sparkling-water-as-a-spark-package

How can I install a 3rd party package in Databricks? Thanks!

pyspark azure-databricks databricks
1 Answers

To make third-party or locally-built code available to notebooks and jobs running on your clusters, you can install a library. Libraries can be written in Python, Java, Scala, and R. You can upload Java, Scala, and Python libraries and point to external packages in PyPI, Maven, and CRAN repositories.

Steps to install third party libraries:

Method1: Using Maven

Step1: Create Databricks Cluster.

Step2: Select the cluster created.

Step3: Select Libraries => Install New => Select Library Source = "Maven" => Coordinates => Search Packages => Select Maven Central => Search for the package required. Example: (ai.h20) => Select the version (sparkling-water-package_2.11:3.30.0.6-1-2.4) required => Install

enter image description here

Method2: Using PYPI

Step1: Create Databricks Cluster.

Step2: Select the cluster created.

Step3: Select Libraries => Install New => Select Library Source = "PyPI" => Coordinates => Enter Packages => h2o-pysparkling-2.4 => Install

enter image description here

Method3: Using pip install

%sh
pip install h2o-pysparkling-2.4==3.30.0.6-1

enter image description here

For more details, refer "Azure Databricks - libraries".

user_81884330
Related