I have been trying to wrap my head around pyarrow for a while, reading their documentation but I still feel like I have not been able to grasp it in it's entirety. I saw their depcrecated method of serialization for arbitrary python objects, but since it's deprecated I was wondering what the correct way is to save for example a list of objects or an arbitrary python object in general?
When do you want to bother using pyarrow as well?
PyArrow is python binding for (Apache) Arrow. Arrow is a cross-language specification that describes how to store columnar data in memory. It serves as the internals of data processing applications & libraries, allowing them to efficiently work with large tabular datasets.
When do you want to bother using pyarrow as well?
One simple use case for PyArrow is to convert between Pandas/Numpy/dict and the Parquet file format. So for example, if you had columnar data (eg DataFrames) that you need to share between programs written in different languages, or even programs using different versions of python, a nice way to do this is to save your Pandas/Numpy/dict to a Parquet file (serialisation). This is a much more portable format that, for example, pickle. It also allows you to embed custom metadata in a portable fashion.
