Get the recommended number of topics from an LDA model using pyspark.ml

I trained an LDA model using pyspark to classify texts by topics, trying different K values. However, to validate the selected K, I want use this aproach evaluate-topic-model-in-python-latent-dirichlet-allocation-lda But, with spark.ml, I dont know how get the equivalent gensim CoherenceModel.

The dataframe looks like this:

tokenizedText.show(truncate=True, n=5)

+------------+--------------------+
|          ID|              Tokens|
+------------+--------------------+
|0000qaqdWUAQ|[limpieza, mala, ...|
|0000qaqe2UAA|[transporte, deja...|
|0000qasxUUAQ|          [correcto]|
|0000qatEJUAY|              [bien]|
|0000qaqwMUAQ|[experiencia, agr...|
+------------+--------------------+

And the basic model is something like this:

from pyspark.ml.feature import IDF, HashingTF, Tokenizer, StopWordsRemover, CountVectorizer
from pyspark.ml.clustering import LDA, LDAModel

counter = CountVectorizer(inputCol="Tokens", outputCol="term_frequency", minDF=5)
counterModel = counter.fit(tokenizedText)   
vectorizedLaw = counterModel.transform(trainingData)

idf = IDF(inputCol="term_frequency", outputCol="tf_idf")
tfidfLaw = idf.fit(vectorizedLaw).transform(vectorizedLaw)

lda = LDA(k=7, maxIter=50, featuresCol="tf_idf", seed=1234)
model = lda.fit(tfidfLaw)

And I get:

model.logLikelihood(tfidfLaw)
Out[295]: -17745244.739330653

model.logPerplexity(tfidfLaw)
Out[296]: 7.63661972904619

Using gensim and follow the evaluate-topic-model-in-python-latent-dirichlet-allocation-lda (Compute Model Perplexity and Coherence Score and Hyperparameter Tuning) example, It was not viable due the data size. After long execution, I got this error:

Internal error, sorry. Attach your notebook to a different cluster or restart the current cluster.
java.net.NoRouteToHostException: No route to host
    at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
    at shaded.v9_4.org.eclipse.jetty.io.SelectorManager.doFinishConnect(SelectorManager.java:355)
    at shaded.v9_4.org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:232)
    at shaded.v9_4.org.eclipse.jetty.io.ManagedSelector.access$1400(ManagedSelector.java:62)
    at shaded.v9_4.org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:543)
    at shaded.v9_4.org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:401)
    at shaded.v9_4.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
    at shaded.v9_4.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
    at shaded.v9_4.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at shaded.v9_4.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    at shaded.v9_4.org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367)
    at shaded.v9_4.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
    at shaded.v9_4.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:914)
    at java.base/java.lang.Thread.run(Thread.java:834)

I am running on Databricks Runtime Version 6.5 ML (includes Apache Spark 2.4.5, Scala 2.11), Driver type: 15.3 GB memory, 2 cores, 1 DBU.

Do you know an appropriate option to obtain the recommended number of topics for the LDA model using pyspark.ml ?, or a workaround to use the Gensim Coherence Score avoiding the execution problem? .