how to save spark dataframe in tfrecords format i tried but giving error

waqar ali 0
Viewed 30

i am trying to save my spark dataframe in tfrecords format , but it's give error.

here is my command ,

df.write.format("tfrecords").mode("overwrite").save()

error message :

Py4JJavaError: An error occurred while calling o1095.save.
: java.lang.ClassNotFoundException: Failed to find data source: tfrecords. Please find packages at http://spark.apache.org/third-party-projects.html

spark version 2.4.5

tensorflow pyspark apache-spark python-3.x
1 Answers

You need to make sure that you run your Spark Job with the spark-tfrecord JAR . The GitHub repository provides some instructions on how to do that.

Robert Kossendey0
Related