BigQuery Storage Write API with C#

user_1353959 0
Viewed 433

I recently discovered that version 2.5.0 of the Google.Cloud.BigQuery.Storage.V1 library now supports the Storage Write API as an alternative to the legacy streaming API (i.e. insertAll):

https://cloud.google.com/dotnet/docs/reference/Google.Cloud.BigQuery.Storage.V1/latest/history

My initial impression is that the C# libraries are similar to the Python example where a compiled protobuf definition is used to initialize the schema definition. I was hoping to find an example closer to what is provided for the Java client where a JsonWriterStream is provided:

https://cloud.google.com/bigquery/docs/write-api#write_data_using_the_jsonstreamwriter

I would really like to use the BigQuery Storage Write API and would like to know if there are any examples of using the API that matches the simplicity of the legacy streaming API?

google-bigquery .net-core c#
1 Answers

There is currently no manual layer written on top of the C# client. Java is the only language currently that supports JSON ingestion. Also, even if manual layer is added for C#, it most likely won't be as easy to use as insertAll (legacy streaming API) since the Write API is a fundamentally different API that supports exactly-once semantic (insertAll is best-effort deduplication) and batching (you have to use Load instead of insertAll for batching).

Here is an article on the overview of the Write API: https://cloud.google.com/blog/topics/developers-practitioners/bigquery-write-api-explained-overview-write-api

More content will be available to talk about the fundamentals of the API.

user_131047940
Related