Azure Data Factory - skipped rows with empty collection reference when flattening JSON

user_2745895 0
Viewed 608

I am trying to serialize the following JSON to Parquet using ADF Copy Activity:

[
    {
        "mac": "06A1E8A75834",
        "timestamp": "2020-11-06T00:00:00+02:00",
        "floor_number": 2,
        "x": 300.00,
        "y": 350.00,
        "located_inside": true,
        "zones": [
            {
                "zone_map_name": "sections",
                "zone_name": "z_1_"
            }
        ]
    },
    {
        "mac": "06A1E8A75835",
        "timestamp": "2020-11-06T00:00:00+02:00",
        "floor_number": 2,
        "x": 300.00,
        "y": 300.00,
        "located_inside": true,
        "zones": []
    }
]

However in some cases the zones array is empty and the rest of the record is skipped. The expected behavior is when the array is empty - the zone_name and zone_map_name are empty.

The mapping looks like this: mapping

Is there a way to tell ADF how to treat empty collection reference as empty values in rows?

azure-data-factory arrays json
1 Answers

As we know, ADF will skipped rows with empty collection reference when flattening JSON.
When we uncheck that box. enter image description here

ADF will copy all the items to destination, like this:

[{"mac":"06A1E8A75834","timestamp":"2020-11-06T00:00:00+02:00","floor_number":2,"x":300.0,"y":350.0,"located_inside":true,"zone_map_name":"sections","zone_name":"z_1_"}
,{"mac":"06A1E8A75835","timestamp":"2020-11-06T00:00:00+02:00","floor_number":2,"x":300.0,"y":300.0,"located_inside":true}
]
user_139794870
Related