Here are the first 5 rows of a dataframe results_df, that we are trying to insert into an empty MongoDB collection using db['our-empty-collection'].insert_many(results_df.to_dict('records')). Presumably because of the last competitionIds column, which is an array of ints, we are getting the error: cannot encode object: array([], dtype=int64), of type: <class 'numpy.ndarray'>. Some observations:
- The
insert_manyis actually working from my local machine, but failing in our docker container. - On my local machine,
pymongo.__version__ == '3.10.1', andpd.__version__ == '0.25.3' - In the docker container,
pymongo.__version__ == '3.11.1', andpd.__version__ == '0.25.3'
...so the only difference in library version is that the local machine, where the insert is working, has a slightly older version of pymongo. Perhaps there is an issue with the int64 types inside the array?, because Mongo handles nested data just fine and I don't see why an array-column in pandas would cause this issue for the insert?
Edit: in Mongo, competitionIds is an array of int32 values, so perhaps the int64 part, rather than the array part, is the issue. Still odd tho to have the local machine / docker container difference.
Edit2: upgraded local pymongo to 3.11.1 to match the docker container's version, ran insert_many() again on my local machine and it worked again on my local machine...
Edit3: numpy.dnarray (on docker container) vs list (on local machine) seems to be the issue... perhaps an issue then with the python GCP / bigquery library version when querying the data. I've updated the local machine bigquery version to 1.25.0 to match the docker-container's version, however still getting the different types for the competitionIds column.

