gcp pubsub to bigquery subscription

Viewed 17

I am trying to publish some data ( in avro schema with JSON encoding) to gcp pubsub and i have enable bigquery subscription so that data will auto route to BQ table.

Avro Schema:

{
  "type": "record",
  "name": "Sample",
  "fields": [
    {
      "name": "abc",
      "type": [
        "null",
        "string"
      ],
      "default": null
    }
  ]
}

BigQuery Table:

Sample_Table(abc: Integer marked as NULLABLE)

When below data is published to topic, then no data is written to BigQuery Table

{
  "abc": {
    "string": "testdata"
  }
}

However, when i publish null values to topic as below, then null is inserted into BigQuery table.

{
  "abc": null
}

Seems like BigQuery subscription is not recognizing the json structure internally?

Can someone please point me to something I am missing here?

1 Answers

This is a known issue with Avro schemas that have a nullable type where the messages are encoded via JSON. You can track the progress on the fix in the issue tracker.

Related