Have data like below FT which is in list with dict format
[{
"DataObject": "DEMURRAGE_AND_DETENTION_AGREEMENT_LINE",
"DemurrageAndDetentionAgreementLineNumber": "001",
"AgreementLineType": {
"AgreementLineTypeCode": "DDFT",
"AgreementLineTypeName": "Demurrage and Detention freetime"
},
"FreeTime": "0022",
"FreeTimeUnit": "DAY",
"CalculationEventTriggers": [
"GATEOUTF"
],
"PricingParameters": [
{
"PricingParameterName": "Equipment_Group",
"PricingParameterValue": "D2222",
"PricingParameterDescription": ".."
},
{
"PricingParameterName": "operator",
"PricingParameterValue": "SCL",
"PricingParameterDescription": ".."
},
{
"PricingParameterName": "commodity",
"PricingParameterValue": "003902",
"PricingParameterDescription": ".."
}
]
}
]
Have data like below RT which is in list with dict format
[{
"DataObject": "DEMURRAGE_AND_DETENTION_AGREEMENT_LINE",
"DemurrageAndDetentionAgreementLineNumber": "002",
"AgreementLineType": {
"AgreementLineTypeCode": "DDRT",
"AgreementLineTypeName": "Demurrage and Detention Rates"
},
"PricingParameters": [
{
"PricingParameterName": "Equipment_Group",
"PricingParameterValue": "D3333",
"PricingParameterDescription": ".."
},
{
"PricingParameterName": "operator",
"PricingParameterValue": "SCL",
"PricingParameterDescription": ".."
},
{
"PricingParameterName": "commodity",
"PricingParameterValue": "003902",
"PricingParameterDescription": ".."
}
],
"Charges": [
{
"StartDay": "0001",
"EndDay": "9999",
"ChargeAmount": "00000001000.00"
}
]
}
]
created function
def get_list_ft_rt(ft,rt): return str(ft + rt)
schema for expected column output:
ft_rt_schema_new = ArrayType(StructType([
StructField('DataObject',StringType(),False),
StructField('DemurrageAndDetentionAgreementLineNumber',StringType(),False),
StructField('AgreementLineType',StructType([StructField('AgreementLineTypeCode',StringType(),False),
StructField('AgreementLineTypeName',StringType(),False)]),False),
StructField('FreeTime',StringType(),False),
StructField('FreeTimeUnit',StringType(),False),
StructField('CalculationEventTriggers',ArrayType(StringType(),False),False),
StructField('PricingParameters',ArrayType(StructType([
StructField('PricingParameterName',StringType(),False),StructField('PricingParameterValue',StringType(),False),StructField('PricingParameterDescription',StringType(),False)]),False),False),
StructField('Charges',ArrayType(StructType([StructField('StartDay',StringType(),False),
StructField('EndDay',StringType(),False),StructField('ChargeAmount',StringType(),False)]),False),False)
]),False)
Created UDF to call function
func_ft_rt_udf = udf(lambda ft,rt: get_list_ft_rt(ft,rt), ft_rt_schema_new)
created dataframe called this function and schema also created expected format
lt_online_ft_rt_0 = lt_online_ft_rt.withColumn('DemurrageAndDetentionAgreementLines',func_ft_rt_udf(col('DemurrageAndDetentionAgreementLines_ft'),col('DemurrageAndDetentionAgreementLines_rt')))
When i display this dataframe it throws error.
PythonException: 'ValueError: Unexpected tuple '[' with StructType'.
with similar error there are few blogs but its not matching our requirement.
expected output for the new column as below
[{
"DataObject": "DEMURRAGE_AND_DETENTION_AGREEMENT_LINE",
"DemurrageAndDetentionAgreementLineNumber": "001",
"AgreementLineType": {
"AgreementLineTypeCode": "DDFT",
"AgreementLineTypeName": "Demurrage and Detention freetime"
},
"FreeTime": "0022",
"FreeTimeUnit": "DAY",
"CalculationEventTriggers": [
"GATEOUTF"
],
"PricingParameters": [
{
"PricingParameterName": "Equipment_Group",
"PricingParameterValue": "D2040",
"PricingParameterDescription": ".."
},
{
"PricingParameterName": "operator",
"PricingParameterValue": "SCL",
"PricingParameterDescription": ".."
},
{
"PricingParameterName": "commodity",
"PricingParameterValue": "003902",
"PricingParameterDescription": ".."
}
]
},
{
"DataObject": "DEMURRAGE_AND_DETENTION_AGREEMENT_LINE",
"DemurrageAndDetentionAgreementLineNumber": "002",
"AgreementLineType": {
"AgreementLineTypeCode": "DDRT",
"AgreementLineTypeName": "Demurrage and Detention Rates"
},
"PricingParameters": [
{
"PricingParameterName": "Equipment_Group",
"PricingParameterValue": "D2040",
"PricingParameterDescription": ".."
},
{
"PricingParameterName": "operator",
"PricingParameterValue": "SCL",
"PricingParameterDescription": ".."
},
{
"PricingParameterName": "commodity",
"PricingParameterValue": "003902",
"PricingParameterDescription": ".."
}
],
"Charges": [
{
"StartDay": "0001",
"EndDay": "9999",
"ChargeAmount": "00000001000.00"
}
]
}
]
Really appreciate in advance for help..