Pubsub Subscription error for TIMESTAMP data type: Incompatible schema type

ryan s 0
Viewed 52

I was working on the new bigquery subscription for pubsub and came across an error when trying to create the subscription:

Incompatible schema type for field 'timestamp': field is STRING in the topic schema, but TIMESTAMP in the BigQuery table schema.

However, in the pubsub documentation there is this statement:

When the type in the topic schema is a string and the type in the BigQuery table is TIMESTAMP, DATETIME, DATE, or TIME, then any value for this field in a Pub/Sub message must adhere to the format specified for the BigQuery data type.

As far as I understand, that means it's possible to insert into a TIMESTAMP column in bigquery as long as the string complies with canonical format defined for TIMESTAMP data type. Am I missing something?

For more info, this is the conflicting part of bigquery and pubsub topic schema that I use:

  • Bigquery schema 
    [ 
  {
    "description": "The exact time when this request is being sent. Set as UTC",
    "mode": "REQUIRED",
    "name": "timestamp",
    "type": "TIMESTAMP"
  }
]
  • Pubsub topic schema 
    syntax = "proto2";

message ExampleMessage {
  required string timestamp = 10;
}

Update: removed space in message name for pubsub topic

protocol-buffers google-cloud-pubsub google-bigquery
1 Answers

It looks like the issue isn't with the field timestamp itself, but with the name of the message type, Example Message. The message type name has a space, which should not be permitted. At this time, Pub/Sub's Protocol Buffer schema validation is more permissive than the standard parser and does not catch errors like this. If you change the name to ExampleMessage, it should work.

A fix for this issue has already been prepared. You can follow the progress of the fix in the public bug on issue tracker.

Kamal Aboul-Hosn0
Related