Bad date format change from string to date in Bigquery

user_16050621 0
Viewed 563

Been struggling with some datasets I want to use which have a problem with the date format. Bigquery could not load the files and returned the following error:

Could not parse '4/12/2016 2:47:30 AM' as TIMESTAMP for field date (position 1) starting at location 21 with message 'Invalid time zone: AM'

I have been able to upload the file manually but as strings, and now would like to set the fields back to the proper format, However, I just could not find a way to change the format of the date column from string to proper DateTime format.

Would love to know if this is possible as the file is just too long to be formatted in excel or sheets (as I have done with the smaller files from this dataset).

google-bigquery sql
2 Answers

now would like to set the fields back to the proper format ... from string to proper DateTime format

Use parse_datetime('%m/%d/%Y %r', string_col) to parse datetime out of string

If applied to sample string in your question - you got

enter image description here

user_52219440

As @Mikhail Berlyant rightly said, using the parse_datetime('%m/%d/%Y %r', string_col)

function will convert your badly formatted dates to a standard format as per ISO 8601 accepted by Google Bigquery . the best option will then be to save these query results to a new table on the database in your Bigquery Project.

I had a similar issue. Below is an image of my table which i uploaded with all columns in String format .

unformatted data

Next up was that i applied the following settings to the query below

Query Settings

The Settings below stored the query output to a new table called heartrateSeconds_clean on the same dataset

The Write if empty option is a good option to avoid overwriting the existing raw data or just arbitrarily writing output to a temporary table, except if you are sure you want to do so. Save the settings and proceed to Run your Query.

Settings to store output in a new table

As seen below, the output schema of the new table is automatically updated Updated schema

Below is the new preview of the resulting table

Resultant Table

NB: I did not apply an ORDER BY clause to the Results hence the data is not ordered by any specific column in both versions of the same table. This dataset has over 2M rows.

user_112781760
Related