Convert array of JSON objects to string in pyspark

Question

I have one requirement in which I need to create a custom JSON from the columns returned from one PySpark dataframe. So I wrote one UDF like the below which will return a JSON in String format from UDF for each row.

Parameter "entities" are in the array of JSON format.

def halResponse(entities, admantx, copilot_id): 
  json_resp = "{\"analyzedContent\": {"+json.dumps(entities)+"}}"
  return json_resp

But in the response, I am not getting proper JSON i.e instead of proper key: value pair, I am just getting values(actual values replace with * for security purpose), not key and value.

Find the sample response:

  "analyzedContents": [
    {
      "entities": [
        [
          "******",
          *,
          *********,
          [
            [
              "***********",
              "***********",
              "***********",
              [
                "*****************"
              ],
              **********
            ]
          ],
          "**************"
        ]
      ]
    }
  ]
}

Please help me to resolve this issue. After fixing, I should get the below sample response

  "analyzedContents": [
    {
      "entities": [
        [
          "key":******",
          "key":*,
          "key":*********,
          [
            [
              "key":"***********",
              "key":"***********",
              "key":"***********",
              [
                "key":"*****************"
              ],
              "key":**********
            ]
          ],
          "key":"**************"
        ]
      ]
    }
  ]
}

user_14165730 · Answer

Try this without using an UDF:

import pyspark.sql.functions as F

df2 = df.withColumn(
    'response',
    F.concat(
        F.lit("{\"analyzedContent\": {"),
        F.to_json(F.col("entities")),
        F.lit("}}")
    )
)

Convert array of JSON objects to string in pyspark

1 Answers