How do I parse a GeoJSON shapefile into a dataset with one row per Feature?

user_14320825 0
Viewed 99

I'm working on a project and need to parse a GeoJSON shapefile of flight route airspace in the US. The data is coming from the FAA open data portal: https://adds-faa.opendata.arcgis.com/datasets/faa::route-airspace/about

There seems to be some relevant documentation at /workspace/documentation/product/geospatial-docs/vector_data_in_transforms where it mentions:

A typical Foundry Ontology pipeline for geospatial vector data may include the following steps:
- Convert into rows with GeoJSON representation of the shape for each feature.

However there isn't actually any guidance on how to go about doing this when the source is a single GeoJSON file with a FeatureCollection and the desired output is a dataset with one row per Feature in the collection.

Anyone have a code snippet for accomplishing this? Seems like a pretty generic task in Foundry.

foundry-code-repositories palantir-foundry
1 Answers

I typically do something like this:

import json

with open('Route_Airspace.geojson', 'r') as f:
    data = json.load(f)

rows = []
for feature in data['features']:
    row = {
        'geometry': json.dumps(feature['geometry']),
        'properties': json.dumps(feature['properties']),
        'id': feature['properties']['OBJECTID']
    }
    rows.append(row)

Note you can leave out the properties, but I like to keep them in this step in case I need them later. Also note this is a good place to set each row's primary key as well (the Features in this dataset have an OBJECTID property, but this may vary)

This rows list can then be used to initialise a Pandas dataframe:

import pandas as pd
df = pd.DataFrame(rows)

or a Spark dataframe (*assuming you're doing this within a transform):

df = ctx.spark_session.createDataFrame(rows)

The resulting dataframes will have one row per Feature, where that feature's shape is contained within the geometry column.

Full example within transform:

from transforms.api import transform, Input, Output
import json

@transform(
    out=Output('Path/to/output'),
    source_df=Input('Path/to/source'),
)
def compute(source_df, out, ctx)
    with source_df.filesystem().open('Route_Airspace.geojson', 'r') as f:
        data = json.load(f)

    rows = []
    for feature in data['features']:
        row = {
            'geometry': json.dumps(feature['geometry']),
            'properties': json.dumps(feature['properties']),
            'id': feature['properties']['OBJECTID']
        }
        rows.append(row)
    
    df = ctx.spark_session.createDataFrame(rows)

    out.write_dataframe(df)

Note that for this to work your GeoJSON file needs to be uploaded into a "dataset without a schema" so the raw file becomes accessible via the FileSystem API.

user_34266500
Related