How to get AWS Glue Schema Registry schema definition using boto3?

user_7252531 0
Viewed 1041

My goal is to receive csv files in S3, convert them to avro, and validate them against the appropriate schema in AWS.

I created a series of schemas in AWS Glue Registry based on the .avsc files I already had:

{
    "namespace": "foo",
    "type": "record",
    "name": "bar.baz",
    "fields": [
        {
            "name": "column1",
            "type": ["string", "null"]
        },
        {
            "name": "column2",
            "type": ["string", "null"]
        },
        {
            "name": "column3",
            "type": ["string", "null"]
        }
    ]
}

But once I try and pull the schemas from Glue the API doesn't seem to provide definition details:

glue = boto3.client('glue')
glue.get_schema(
    SchemaId={
        'SchemaArn': schema['SchemaArn']
    }
)

returns:

{
    'Compatibility': 'BACKWARD',
    'CreatedTime': '2021-08-11T21:09:15.312Z',
    'DataFormat': 'AVRO',
    'LatestSchemaVersion': 2,
    'NextSchemaVersion': 3,
    'RegistryArn': '[my-registry-arn]',
    'RegistryName': '[my-registry-name]',
    'ResponseMetadata': {
        'HTTPHeaders': {
            'connection': 'keep-alive',
            'content-length': '854',
            'content-type': 'application/x-amz-json-1.1',
        },
        'HTTPStatusCode': 200,
        'RetryAttempts': 0,
    },
    'SchemaArn': '[my-schema-arn]',
    'SchemaCheckpoint': 2,
    'SchemaName': '[my-schema-name]',
    'SchemaStatus': 'AVAILABLE',
    'UpdatedTime': '2021-08-11T21:09:17.312Z',
}

Is there a way to programmatically retrieve the Glue Schema Registry definitions for a schema? Or am I taking the wrong approach here with what I'm trying to do?

aws-glue boto3 amazon-web-services python
2 Answers

After some more digging I found the somewhat confusingly named get_schema_version() method that I had been overlooking which returns the SchemaDefinition:

{
    'SchemaVersionId': 'string',
    'SchemaDefinition': 'string',
    'DataFormat': 'AVRO'|'JSON',
    'SchemaArn': 'string',
    'VersionNumber': 123,
    'Status': 'AVAILABLE'|'PENDING'|'FAILURE'|'DELETING',
    'CreatedTime': 'string'
}
user_72525310

#If you are using glue schema registry:

session = boto3.Session( region_name='us-east-1')

glue_client = session.client('glue')
#glue = boto3.client('glue')
response = glue_client.list_registries(
    MaxResults=23
)


schema_message = glue_client.get_schema_version(
    SchemaId={
        'SchemaName': 'string',
        'RegistryName': 'string'
    },
    SchemaVersionNumber={
        'LatestVersion': True
    }
)
print(schema_message['SchemaDefinition'])
user_30829280
Related