PubSub & BigQuery subscription using topic schema

user_1037965 0
Viewed 545

With the new direct PubSub to BigQuery announced July 2022, we can now create BigQuery subscriptions that get data pushed from PubSub directly.

When it comes to timestamp fields, how are they mapped to BigQuery? Things I tried:

  • Enable "use schema" in PubSub and create a BQ table with a created_at: TIMESTAMP

  • The documentation doesn't seem to specify a timestamp mapping for Protobuf, so ignoring this.

  • When using Avro Logical fields, the error I'm getting when creating a subscription is like Incompatible schema type for field created_at: expected INT64, got TIMESTAMP.

Example of PubSub topic schemas tried:

{
  "type": "record",
  "name": "Avro",
  "fields": [
    {
      "name": "created_at",
      "type": {
        "type": "<long|int>",
        "logicalType": "timestamp-<millis|micros>"
      }
    }
  ]
}

What's the correct way to create a PubSub schema for Timestamp fields?

google-cloud-pubsub google-bigquery
1 Answers

Right now, logical types are not supported with BigQuery subscriptions. The documentation is being updated to reflect this fact and support is being added as we speak. Expect it to be ready by the end of Q3 2022. For now, the TIMESTAMP logical type would map to the underlying type, which would be INT64 in this case.

user_11212880
Related