PythonException: 'ValueError: Unexpected tuple '[' with StructType' not able add two lists into single in pysaprk

Viewed 55

Have data like below FT which is in list with dict format

  [{
    "DataObject": "DEMURRAGE_AND_DETENTION_AGREEMENT_LINE",
    "DemurrageAndDetentionAgreementLineNumber": "001",
    "AgreementLineType": {
      "AgreementLineTypeCode": "DDFT",
      "AgreementLineTypeName": "Demurrage and Detention freetime"
    },
    "FreeTime": "0022",
    "FreeTimeUnit": "DAY",
    "CalculationEventTriggers": [
      "GATEOUTF"
    ],
    "PricingParameters": [
      {
        "PricingParameterName": "Equipment_Group",
        "PricingParameterValue": "D2222",
        "PricingParameterDescription": ".."
      },
      {
        "PricingParameterName": "operator",
        "PricingParameterValue": "SCL",
        "PricingParameterDescription": ".."
      },
      {
        "PricingParameterName": "commodity",
        "PricingParameterValue": "003902",
        "PricingParameterDescription": ".."
      }
    ]
    }
    ]

Have data like below RT which is in list with dict format

[{
"DataObject": "DEMURRAGE_AND_DETENTION_AGREEMENT_LINE",
"DemurrageAndDetentionAgreementLineNumber": "002",
"AgreementLineType": {
  "AgreementLineTypeCode": "DDRT",
  "AgreementLineTypeName": "Demurrage and Detention Rates"
},
"PricingParameters": [
  {
    "PricingParameterName": "Equipment_Group",
    "PricingParameterValue": "D3333",
    "PricingParameterDescription": ".."
  },
  {
    "PricingParameterName": "operator",
    "PricingParameterValue": "SCL",
    "PricingParameterDescription": ".."
  },
  {
    "PricingParameterName": "commodity",
    "PricingParameterValue": "003902",
    "PricingParameterDescription": ".."
  }
],
"Charges": [
  {
    "StartDay": "0001",
    "EndDay": "9999",
    "ChargeAmount": "00000001000.00"
  }
]
}
]

created function

def get_list_ft_rt(ft,rt): return str(ft + rt)

schema for expected column output:

ft_rt_schema_new = ArrayType(StructType([
                            StructField('DataObject',StringType(),False),
                            StructField('DemurrageAndDetentionAgreementLineNumber',StringType(),False),
                            StructField('AgreementLineType',StructType([StructField('AgreementLineTypeCode',StringType(),False),
                            StructField('AgreementLineTypeName',StringType(),False)]),False),
                            StructField('FreeTime',StringType(),False),
                            StructField('FreeTimeUnit',StringType(),False),
                            StructField('CalculationEventTriggers',ArrayType(StringType(),False),False),
                            StructField('PricingParameters',ArrayType(StructType([
                            StructField('PricingParameterName',StringType(),False),StructField('PricingParameterValue',StringType(),False),StructField('PricingParameterDescription',StringType(),False)]),False),False),
                            StructField('Charges',ArrayType(StructType([StructField('StartDay',StringType(),False),
                            StructField('EndDay',StringType(),False),StructField('ChargeAmount',StringType(),False)]),False),False)
                    ]),False)

Created UDF to call function

func_ft_rt_udf = udf(lambda ft,rt: get_list_ft_rt(ft,rt), ft_rt_schema_new)

created dataframe called this function and schema also created expected format

lt_online_ft_rt_0 = lt_online_ft_rt.withColumn('DemurrageAndDetentionAgreementLines',func_ft_rt_udf(col('DemurrageAndDetentionAgreementLines_ft'),col('DemurrageAndDetentionAgreementLines_rt')))

enter image description here

When i display this dataframe it throws error.

PythonException: 'ValueError: Unexpected tuple '[' with StructType'.

with similar error there are few blogs but its not matching our requirement.

expected output for the new column as below

 [{
    "DataObject": "DEMURRAGE_AND_DETENTION_AGREEMENT_LINE",
    "DemurrageAndDetentionAgreementLineNumber": "001",
    "AgreementLineType": {
      "AgreementLineTypeCode": "DDFT",
      "AgreementLineTypeName": "Demurrage and Detention freetime"
    },
    "FreeTime": "0022",
    "FreeTimeUnit": "DAY",
    "CalculationEventTriggers": [
      "GATEOUTF"
    ],
    "PricingParameters": [
      {
        "PricingParameterName": "Equipment_Group",
        "PricingParameterValue": "D2040",
        "PricingParameterDescription": ".."
      },
      {
        "PricingParameterName": "operator",
        "PricingParameterValue": "SCL",
        "PricingParameterDescription": ".."
      },
      {
        "PricingParameterName": "commodity",
        "PricingParameterValue": "003902",
        "PricingParameterDescription": ".."
      }
    ]
  },
  {
    "DataObject": "DEMURRAGE_AND_DETENTION_AGREEMENT_LINE",
    "DemurrageAndDetentionAgreementLineNumber": "002",
    "AgreementLineType": {
      "AgreementLineTypeCode": "DDRT",
      "AgreementLineTypeName": "Demurrage and Detention Rates"
    },
    "PricingParameters": [
      {
        "PricingParameterName": "Equipment_Group",
        "PricingParameterValue": "D2040",
        "PricingParameterDescription": ".."
      },
      {
        "PricingParameterName": "operator",
        "PricingParameterValue": "SCL",
        "PricingParameterDescription": ".."
      },
      {
        "PricingParameterName": "commodity",
        "PricingParameterValue": "003902",
        "PricingParameterDescription": ".."
      }
    ],
    "Charges": [
      {
        "StartDay": "0001",
        "EndDay": "9999",
        "ChargeAmount": "00000001000.00"
      }
    ]
  }
]

Really appreciate in advance for help..

1 Answers

For me issue got resolved by keeping same schema between FT and RT columns then able to query data frame.

Related