Cannot insert pandas dataframe to mongo - cannot encode object: array([], dtype=int64), of type: <class 'numpy.ndarray'>

Viewed 287

enter image description here

Here are the first 5 rows of a dataframe results_df, that we are trying to insert into an empty MongoDB collection using db['our-empty-collection'].insert_many(results_df.to_dict('records')). Presumably because of the last competitionIds column, which is an array of ints, we are getting the error: cannot encode object: array([], dtype=int64), of type: <class 'numpy.ndarray'>. Some observations:

  • The insert_many is actually working from my local machine, but failing in our docker container.
  • On my local machine, pymongo.__version__ == '3.10.1', and pd.__version__ == '0.25.3'
  • In the docker container, pymongo.__version__ == '3.11.1', and pd.__version__ == '0.25.3'

...so the only difference in library version is that the local machine, where the insert is working, has a slightly older version of pymongo. Perhaps there is an issue with the int64 types inside the array?, because Mongo handles nested data just fine and I don't see why an array-column in pandas would cause this issue for the insert?

Edit: in Mongo, competitionIds is an array of int32 values, so perhaps the int64 part, rather than the array part, is the issue. Still odd tho to have the local machine / docker container difference.

Edit2: upgraded local pymongo to 3.11.1 to match the docker container's version, ran insert_many() again on my local machine and it worked again on my local machine...

Edit3: numpy.dnarray (on docker container) vs list (on local machine) seems to be the issue... perhaps an issue then with the python GCP / bigquery library version when querying the data. I've updated the local machine bigquery version to 1.25.0 to match the docker-container's version, however still getting the different types for the competitionIds column.

enter image description here

0 Answers
Related