How to write to yaml file in human readable format?

user_913098 0
Viewed 932

I have some data types I need to write:

a. A list of numpy arrays, e.g. [ndarray, ndarray, ndarray] of different sizes.

b. Any arbitrary numpy array, e.g. np.zeros((5,6)), np.randn((76,2)) and so on.

c. Any other future datatype that hasn't occurred to me yet.

Requirements:

  1. I need a single function to be able to save all those data types, with no specific handling, and with future compatibility for type c stated above.

  2. I also need the output file dump in human readable format.

So far, I was only able to achieve requirement 1 with either YAML or pickle, both of which with binary files, i.e. not human readable.

@staticmethod
def _read_with_yaml(path):
    with open(path, 'r') as stream:
        return yaml.load(stream)

@staticmethod
def _write_with_yaml(path, obj):
    with io.open(path, 'w+', encoding='utf8') as outfile:
        yaml.dump(obj, outfile, default_flow_style=False, allow_unicode=True)

This example code outputs non-human-readable files, but works for the data types I have.

Is there a way to achieve both requirements?

pickle serialization yaml file python
1 Answers

No, your requirements cannot be satisfied.

You already have one function yaml.dump() that saves all those data types. As you noticed it doesn't do so in a very readable way for numpy data structures. This is caused by numpy not having dumping routines for their special data structure, instead falling back to the, not-so-readable, default !python.... tagged dump of the datastructure. Now you (or the YAML or Numpy package maintainers) can provide, special routines for those objects that dump in a more readable format, so that could be covered. You can make the representer in your YAML library more intelligent and get more readable output for Numpy datastructures without touching Numpy classes.

But you want this for all future datatypes, and IMO a variation of Gödel's theorem applies: even if the YAML library is extended so that it covers all known cases and dumps them in a readable way, there will always be new datastructures, especially in C based extensions (like Numpy), that cannot be represented in a readable way without extra work.

So because of your

Any other future unknown datatype that hasn't occurred to me yet.

premise, this is not just a lot of difficult work, but impossible.

user_13079050
Related