How to cast pyarrow timestamp dtype to time64 type?

user_3297613 0
Viewed 743

I'm trying to cast pyarrow timestamp type of time64 type. But it's showing cast error.

import pyarrow as pa
from datetime import datetime

dt = datetime.now()
table = pa.Table.from_pydict({'ts': pa.array([dt, dt])})
new_schema = table.schema.set(0, pa.field('ts', pa.time64('us')))
table.schema
# ts: timestamp[us]
new_schema
# ts: time64[us]

table.cast(new_schema)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/table.pxi", line 1329, in pyarrow.lib.Table.cast
  File "pyarrow/table.pxi", line 277, in pyarrow.lib.ChunkedArray.cast
  File "/home/inspiron/.virtualenvs/par/lib/python3.7/site-packages/pyarrow/compute.py", line 243, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 446, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 275, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 105, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from timestamp[us] to time64 using function cast_time64

Is there any way to make this casting possible?

apache-arrow pyarrow python
1 Answers

time64[us] is a time of day. It represents the number of microseconds since midnight. It is not tied to any specific date and cannot be converted to a timestamp.

The Arrow docs are a bit sparse but the parquet docs explain better:

TIME

TIME is used for a logical time type without a date with millisecond or microsecond precision. The type has two type parameters: UTC adjustment (true or false) and unit (MILLIS or MICROS, NANOS).

TIME with unit MILLIS is used for millisecond precision. It must annotate an int32 that stores the number of milliseconds after midnight.

TIME with unit MICROS is used for microsecond precision. It must annotate an int64 that stores the number of microseconds after midnight.

TIME with unit NANOS is used for nanosecond precision. It must annotate an int64 that stores the number of nanoseconds after midnight.

The sort order used for TIME is signed.

user_2026940
Related