Actually I am trying to write Spark Dataframe to Json format. I want to avoid 0 value attribute in json dump therefore trying to set the value in all columns with zero value to None/NULL.
Note: Since I am using pivot method to dynamically create columns, I cannot do with at each columns level.
+----------------------+---------------+----------+------------+-------------+--------------------+--------------+
|ID |T1 |T2 |T3 |T4 |T5 |DT |
+----------------------+---------------+----------+------------+-------------+--------------------+--------------+
|929916248484355237|0.000000 |0.0 |0E-10 |20.0000000000|20.0 |20220916120738|
|772216248474350399|0.880000 |0.0 |1.4808200000|0E-10 |2.36082 |20220916120738|
|772216248474350399|0.000000 |0.0 |0E-10 |20.0000000000|20.0 |20220916120738|
|075616248464351729|0.010000 |0.0 |0.0127000000|0E-10 |0.022699999999999998|20220916120738|
|915716248424355578|0.010000 |0.0 |0.0127000000|0E-10 |0.022699999999999998|20220916120738|
|277016248484357606|0.000000 |0.0 |0E-10 |20.0000000000|20.0 |20220916120738|
|739516248574350647|0.000000 |0.0 |0E-10 |20.0000000000|20.0 |20220916120738|
|915716248424355578|0.000000 |0.0 |0E-10 |20.0000000000|20.0 |20220916120738|
|075616248464351729|0.000000 |0.0 |0E-10 |20.0000000000|20.0 |20220916120738|
|756216248594352650|0.000000 |0.0 |0E-10 |20.0000000000|20.0 |20220916120738|
+----------------------+---------------+----------+------------+-------------+--------------------+--------------+
Expected Result:
+----------------------+---------------+----------+------------+-------------+--------------------+--------------+
|ID |T1 |T2 |T3 |T4 |T5 |DT |
+----------------------+---------------+----------+------------+-------------+--------------------+--------------+
|929916248484355237|null |0.0 |null |20.0000000000|20.0 |20220916120738|
|772216248474350399|0.880000 |0.0 |1.4808200000|null |2.36082 |20220916120738|
|772216248474350399|null |0.0 |null |20.0000000000|20.0 |20220916120738|
|075616248464351729|0.010000 |0.0 |0.0127000000|null |0.022699999999999998|20220916120738|
|915716248424355578|0.010000 |0.0 |0.0127000000|null |0.022699999999999998|20220916120738|
|277016248484357606|null |0.0 |null |20.0000000000|20.0 |20220916120738|
|739516248574350647|null |0.0 |null |20.0000000000|20.0 |20220916120738|
|915716248424355578|null |0.0 |null |20.0000000000|20.0 |20220916120738|
|075616248464351729|null |0.0 |null |20.0000000000|20.0 |20220916120738|
|756216248594352650|null |0.0 |null |20.0000000000|20.0 |20220916120738|
+----------------------+---------------+----------+------------+-------------+--------------------+--------------+
I tried with this and it returns as expected
cols = [when(~col(x).isin(0), col(x)).alias(x) for x in df.columns]
df = df.select(*cols)
Any other best approach is highly appreciated.