I have the following dataframe
df = spark.createDataFrame([['<?xml version="1.0" encoding="UTF-8" standalone="yes"?>\
<ns1:entities>\
<ns1:entity ns1:type="PARENT">\
<ns1:attribute ns1:name="PARENT_ID">12345</ns1:attribute>\
<ns1:entity ns1:instance="207" ns1:type="CHILD" ns1:id="P1">\
<ns1:attribute ns1:name="CHILD_ID">2071</ns1:attribute>\
<ns1:attribute ns1:name="QUESTION">One</ns1:attribute>\
<ns1:attribute ns1:name="QUESTION1981"/>\
<ns1:attribute ns1:name="ANS1981"/>\
<ns1:attribute ns1:name="TEST"/>\
<ns1:link ns1:name="QUESTION1981"/>\
</ns1:entity>\
</ns1:entity>\
</ns1:entities>']], ['visitors'])
My Goal is to replace all ns1:attributes which are empty tags with null value
i.e. I want to search and replace like this
search <ns1:attribute ns1:name="QUESTION1981"/> and replace with
<ns1:attribute ns1:name="QUESTION1981">null</ns1:attribute>
So I am looking for a generic solution ,where it will only search within ns1:attribute and replace i.e my outcome should be
<ns1:attribute ns1:name="QUESTION1981">null</ns1:attribute>
<ns1:attribute ns1:name="ANS1981">null</ns1:attribute>
<ns1:attribute ns1:name="TEST">null</ns1:attribute>
Is there a generic way to do itc,so far I have tried below
df=df.withColumn('visitors',regexp_replace('visitors','<ns1:attribute\s*\w+\d+:\w+="\w+"\s*/>','<ns1:attribute>null</ns1:attribute>'))
display(df)
Search part working properly,but I am struggling with replace