The job is submitted and run successfully. But there is just no data in the bucket. How should I resolve it?
df = spark.createDataFrame([["Amy", "lily", 12], ["john", "tom", 34]]).toDF(*["first_name", "last_name", "age"])
df.write.format("parquet").partitionBy("age").option("path", "gs://my_bucket/my_table")
The code from the question configures the write operation, but never triggers the write itself.
In order to actually trigger the write operation you need to call one of the save functions in the Writer interface.
For example, following will do the job:
df.write.format("parquet").partitionBy("age").option("path", "gs://my_bucket/my_table").save()
or:
df.write.format("parquet").partitionBy("age").save("gs://my_bucket/my_table")
or even:
df.write.partitionBy("age").parquet("gs://my_bucket/my_table")
Mode details:
df.write returns an instance of a DataFrameWriter; here is the API: https://spark.apache.org/docs/2.4.6/api/scala/index.html#org.apache.spark.sql.DataFrameWriter
DataFrameWriter API is consistent, in its spirit, with all other Spark APIs: it is lazy. Nothing is executed unless an action is triggered.
For this, instances of the DataFrameWriter behave similarly to a builder pattern implementation: subsequent calls to the format, option, mode et al. only configure the write operation that may be eventually executed. Once the operation is configured, you can trigger it by calling save or a similar method on this instance.
Similarly, DataFrameWriter also allows you to reuse the write operation multiple times (e.g., configure a base set of options and then call twice to write parquet and csv files, for example; or write to different locations, etc.).