How to run non-spark model training task (using fasttext) efficiently on a databricks cluster?

user_10715700 0
Viewed 136

I want to train some models using fasttext and since it doesn't use spark, it will be running on my driver. The number of training jobs that will be running simultaneously is very large and so is the size of the data. Is there a way to make it run it on different workers or distribute it across workers? Is this the best approach or am I better off using a large single node cluster?

FYI, I am using Databricks. So solutions specific to that are also okay.

fasttext mlflow pyspark databricks apache-spark
1 Answers

You can use Databricks multi-node clusters to run training even for the libraries that are effectively single-node, such as scikit-learn, etc. This is typically done using the HyperOpt library that is bundled together with ML runtimes. You will need to define an objective function, but it's implementation depends on the differences of models. Look into this example that shows how to run different algorithms from scikit-learn.

Alex_Ott_186270
Related