Python writing AVRO timestamp-millis: datum.astimezone(tz=timezones.utc) AttributeError: 'int' object has no attribute 'astimezone'

user_2700344 0
Viewed 1199

Environment: python 3.8.4 under Windows 10.

I am trying to write avro from dictionary.

Dictionary contains timestamp

def get_dict(self):
        return {"msg_header": {...
                                 "msg_timestamp": int(datetime.datetime.timestamp(datetime.datetime.now())),
                               ...},
                ...
                 }

Avro schema:

{
        "name" : "msg_timestamp",
        "type" : {
          "type" : "long",
          "logicalType" : "timestamp-millis"
}

Writing Avro:

writer = avro.io.DatumWriter(schema)
bytes_writer = io.BytesIO()
encoder = avro.io.BinaryEncoder(bytes_writer)
writer.write(get_dict(), encoder) #----------Exception here
raw_bytes = bytes_writer.getvalue()

Exception:

...
line
581, in write_timestamp_millis_long
    datum = datum.astimezone(tz=timezones.utc)  
AttributeError: 'int' object has no attribute 'astimezone'

Any ideas how to fix it? I do not want conversion to UTC. Even if I set system TZ to UTC, it does not help, method astimezone(tz=timezones.utc) is being executed causing exception

avro python-3.x timestamp python
2 Answers

Finally I ended up using fastavro, it provides more verbose exception messages rather than "the datum ... is not an example of the schema...", it works without any issue and process a list of messages.

import fastavro
...
parsed_schema = fastavro.schema.load_schema(schema_path)
...
def get_dict(self):
    return {"msg_header": {...
                             "msg_timestamp": int(datetime.datetime.timestamp(datetime.datetime.now())),
                           ...},
            ...
             }
...

#in a for loop do something and Prepare list of messages 
messages.append(get_dict().copy())
...

bytes_writer = io.BytesIO()
fastavro.writer(bytes_writer, parsed_schema, messages)
raw_bytes = bytes_writer.getvalue()
user_27003440

msg_timestamp is defined as a logical type so that means your dictionary should contain a datetime and the library will automatically convert it to a long when serializing and back to a datetime when deserializing.

So instead of:

"msg_timestamp": int(datetime.datetime.timestamp(datetime.datetime.now()))

You just want to do:

"msg_timestamp": datetime.datetime.now(tz=datetime.timezone.utc)

If instead you want to turn it to an int yourself and not have the library do it, don't use logicalType.

user_61314520
Related