How to store renewed token during API Calls using spark dataframe

user_1757152 0
Viewed 244

I'm currently working on a requirement where we need to bring data from REST API using pyspark. We deal with good amount of data and hence need to do paginating and usually takes good amount of time to retrieve all data. The REST API uses Oauth2.0 and the issued token expires in 15minutes.

I created a UDF to run the API call which is called from dataframe as below. I feed the list of URLs along with access token to the UDF.

df2 = df.withColumn('output', test_udf(col("url"),col("id")))

I'm still trying to figure out how to pass the renewed access token when the initial one expires. I thought of storing the access token in databricks delta table instead of passing through function and update it when it expires but looks like that approach doesn't work. Any thoughts are highly appreciated.

pyspark databricks apache-spark oauth-2.0 rest
1 Answers

You just need to move token generation/renewal functionality into your function, so it will get new token when old is close to expiration, or you get "token expired" error.

epending on your implementation, you can further optimize performance to avoid sending multiple renewal requests, for example, by using global variables or something like, but first try with simple implementation. Please also note that in some cases (like, "normal" Python UDFs), you'll get a separate function call per each row, so some caching could be required.

Alex_Ott_186270
Related