Worker Type and Driver Type Settings in Databricks with ThreadPoolExecutor

I want to run ThreadPoolExecutor() in Databricks for 26 threads. However it times out still after 45min even if I have 26 threads running.

I don't think I have my worker or driver type settings correct in Databricks.

The time to run this code should only take max 20 min if all 26 jobs are working at the same time.

Does anyone know if this is the reason why I cannot run all 26 threads at the same time?

The current cluster settings are:

Here is my code:

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def processAnIntegerNumber(START):
  dbutils.notebook.run(path = "/Users/nickknauer1@gmail.com/Ticker_Price_Prediction",
                                        timeout_seconds = 2700, 
                                        arguments = {"START":START})

listOfNumbers = `["1:500","501:1000","1001:1500","1501:2000","2001:2500","2501:3000","3001:3500","3501:4000","4001:4500","4501:5000","5001:5500","5501:6000","5001:5500","5501:6000","6001:6500","6501:7000","7001:7500","7501:8000","8001:8500","8501:9000","9001:9500","9501:10000","10001:10500","10501:11000","11001:11500","11501:12000"]`
with ThreadPoolExecutor(26) as executor:
  results = executor.map(processAnIntegerNumber, listOfNumbers)

This produces 26 jobs:

Notebook job #854127233194966
Notebook job #598515092856697
Notebook job #698888042699647
Notebook job #521308903687621
Notebook job #833774957866049
Notebook job #967838289186350
Notebook job #1044656684371779
Notebook job #967728381316488
Notebook job #756103818481769
Notebook job #741571723642216
Notebook job #305407888725604
Notebook job #750971338842454
Notebook job #364541218225804
Notebook job #546789683043714
Notebook job #990076496894968
Notebook job #891315472543179
Notebook job #581596503752099
Notebook job #440871497774881
Notebook job #244651017943245
Notebook job #37565644804918
Notebook job #1013093731112037
Notebook job #206008865032730
Notebook job #702612260313649
Notebook job #449829028183619
Notebook job #988510834430189
Notebook job #352412251594948
Notebook job #1009868477781987
Notebook job #253401804409472
Notebook job #725192903682173
Notebook job #247174921415516
Notebook job #468244980716449
Notebook job #619622249572242
Notebook job #623722409320239
Notebook job #1062537954502394
Notebook job #232366090430858
Notebook job #1120897464937076
Notebook job #957077770729470
Notebook job #580409630523552
Notebook job #870069421036075
Notebook job #669465748375934
Notebook job #698596574571091
Notebook job #583143393809739

EDIT: Underlying code within "/Users/nickknauer1@gmail.com/Ticker_Price_Prediction":

For context, I am trying to predict future stock prices for all 12K stocks within NYSE. This list of numbers 1:500, 501:1000, etc. represent the ticker number associated with the ticker name. So for example, in each job I am trying to predict the future price for 500 stocks/tickers. The underlying code to do the model can be found below:

All ML work is done in R.

Step 1: Call 500 stocks/tickers with historical stock prices from Snowflake.

Step 2: Predict future values for each stock price (500 filtered)

data_new = latest_combined_stock_data %>%  
  group_by(TICKER) %>%
  mutate(weekdays = weekdays(ds)) %>% 
  filter(weekdays != "Saturday" & weekdays != "Sunday") %>%
  do(predict(prophet(., daily.seasonality = TRUE, yearly.seasonality = TRUE), 
             filter(make_future_dataframe(prophet(., daily.seasonality = TRUE, yearly.seasonality = TRUE), periods = 14), weekdays(ds) != "Saturday" & weekdays(ds) != "Sunday"))) %>%
  select(ds, TICKER, yhat)
data_new$ds <- as.Date(data_new$ds)

Since there are 12K, I separated it into 26 independent jobs (500 stocks per job) to make it run faster.