What will happen if one or more processes are trying to append data to the table and one or more processes are trying to alter the table at the same time?
Let's say we have a table called db.table(partitioned on band column):
band |name |age
---------------------------
1 abc 18
1 xyz 20
2 aaa 18
2 bbb 19
3 ccc 22
4 ddd 28
Alter command:
spark.sql("alter table db.table add columns (year_of_birth int, salary double, city string)")
Data append:
df = spark.sql("select 5 as band, 'fff' as name, 33 as age")
df.write.partitionBy('band').mode('append').saveAsTable('db.table')
Here df.write and spark.sql(alter command) are executing at the same time by multiple processes. What is the behaviour of spark in this scenario?