When should I use Azure ML Notebooks VS Azure Databricks? Both are competitor products in my opinion

user_5554046 0
Viewed 3755

Pretty self-explanatory question. When should I use Azure ML Notebooks VS Azure Databricks? I feel there’s a great overlap between the two products and one is definitely better marketed than the other..

I’m mainly looking for information concerning datasets sizes and typical workflow. Why should I use Databricks over AzureML if I don’t have a Spark oriented workflow ?

Thanks !

azure-machine-learning-service databricks machine-learning azure
1 Answers

@Nethim, from my pov these are the main difference:

  1. Data Distribution:

    • Azure ML Notebooks are good when you are training with a limited data on single machine. Though Azure ML provides training clusters, the data distribution among the nodes is to be handled in the code.
    • Azure Databricks with its RDDs are designed to handle data distributed on multiple nodes.This is advantageous when your data size is huge.When your data size is small and can fit in a scaled up single machine/ you are using a pandas dataframe, then use of Azure databricks is a overkill

  2. Data Cleaning: Databricks can support a lot of file formats natively and querying and cleaning huge datasets are easy where as this has to be handled custom in AzureML notebooks. This can be done with a aml notebooks but cleaning and writing to stores has to be handled.

  3. Training Both has the capabilities if distributing the training, Databricks provides inbuilt ML algorithms that can act on chunk of data on that node and coordinate with other nodes. Though this can be done on both AzureMachineLearning and Databricks with tf,horovod etc.,

In general(just my opinion), if the dataset is small, aml notebooks is good.If the data size is huge, then Azure databricks is easy for datacleanup and format conversions.Then the training can happen on AML or databricks.Though databricks has a learning curve whereas Azure ML can be easy with the python and pandas.

Thanks.

user_124513800
Related