What is the best way to Serialize/Deserialize Class Objects in Python?

user_16736621 0
Viewed 2022

I am trying to log some data from units that I am working with for long term and short term data collection and storage.

Should I use the json library or the pickle library and then how do I load the units' data from a file, be able to manipulate it, add units, etc., and then save the updated data back to the file?

Here is a detailed explanation of my problem and where I am stuck:

For each unit I have some data in a class object:

unit_1.name = 'unit_1'
unit_1.folder = 'C:\\Users\\Thomas\\Documents\\1'
unit_1.serial_number = 1
unit_1.offsets = [0.00, 0.00, 0.00, 0.00 ...]
unit_1.status = 'Open'
etc

I would like to store these objects for long term storage (so in a file), but I also need to be able to access them from the command line as a class object by doing something like this:

>python
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>from file import unit_log
>unit_log.load()
>unit_1.serial_number
1

and I would like to create new units like this:

>unit_2 = unit_log(2)

which would create the other fields in the init module of the class. After which I could save the units that I created or modified by doing something like:

>unit_log.save()

and it would save all the units to the file. Here is the code for the class:

file.py

class unit_log:
    def __init__ (self, serial_number):
        self.serial_number = serial_number;
        self.unit_name = 'unit_'+str(serial_number)
        self.status = 'Open';
        self.offsets = [];
        self.folder = 'C:\\Users\\Thomas\\Documents\\' + str(serial_number)
    @classmethod
    def load (self):
        #Something that loads the data
    @classmethod
    def save (self):
        #Something that saves the data

My current understanding is that what I am trying to do is called serialization/deserialization and that there are a couple different ways to do this (the big ones being the pickle library and the json library).

My question is: How do I deserialize a json file (or a pickle file) into a bunch of class objects? I am assuming the json file would look something like this:

unit_logs.json
{
    unit_1:{
    {
    "unit_name": "unit_1",
    "serial_number": 1,
    "status": "Open",
    "offsets": [],
    "folder": "C:\\Users\\Thomas\\Documents\\1"
    }
    unit_2...
    unit_3...
    ...
}

and I would load in the file using log = json.load(unit_logs.json). I would like to then have a

for unit in log:
    unit["unit_name"] = unit_log(unit)

but that is not how declaring a variable works.

I believe that the pickle library works similarly but I am less familiar with that one.

Should I use the json library or the pickle library and then how do I load the units' data from a file, be able to manipulate it, add units, etc., and then save the updated data back to the file?

deserialization pickle serialization python json
3 Answers

pickle library is easy to use

Dump an object obj to file:

import pickle

with open(f_path, "wb") as f:
    pickle.dump(obj, f)

Load a pickle file:

with open(f_path, "rb") as f:
    obj = pickle.load(f)

Then you can change obj and write it again to file

In case you want to load the same pickle file several times and have different objects you can just:

N=<number of copies>

with open(f_path, "rb") as f:
    obj_list = [pickle.load(f) for _ in range(N)]

Now you can edit each object in obj_list separately, and if you want to save its state to a pickle file, you use pickle.dump as described above.

So you can have a "template" pickle file to load from, and then edit the objects as you want.

user_150989920

If you want to you use json, just try this:

json.dumps(unit_log_instance.__dict__)
user_150075490

Here is something you might useful. There is a Python package available to smartly construct objects from YAML/JSON/dicts, and is actively being developed and expanded. (full disclosure, I am a co-author of this package, see here)

EDIT:

Now, in your case, you might consider using it as follows:

As a YAML file (filename.yaml) (or JSON), the following could be defined:

name: 'unit_1'
folder: 'C:\\Users\\Thomas\\Documents\\1'
serial_number: 1
offsets: [0.00, 0.00, 0.00, 0.00 ]
status: 'Open'

Loading this as a Python object from file is achieved:

>> python
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

> from pickle_rick import PickleRick
> unit_1 = PickleRick('./filename.yaml')
> unit_1.serial_number
1

Changes can then be made and saved to file again:

> unit_1.serial_number += 41
> unit_1.serial_number
42
> unit_1.to_yaml_file('./filename.yaml')

Or as JSON file.

Now, to have more than one unit, consider a structure as this:

units:
    - name: 'unit_1'
      folder: 'C:\\Users\\Thomas\\Documents\\1'
      serial_number: 1
      offsets: [0.00, 0.00, 0.00, 0.00 ]
      status: 'Open'

    - name: 'unit_2'
      folder: 'C:\\Users\\Thomas\\Documents\\1'
      serial_number: 1
      offsets: [0.00, 0.00 ]
      status: 'Closed'

Then this can be loaded as a list of units:

>> python
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

> from pickle_rick import PickleRick
> all_units = PickleRick('./filename.yaml', deep=True)
> all_units.units[0].serial_number
1

Original post:

Install:

pip install pickle-rick

Use:

Define a YAML or JSON string (or file).

BASIC:
 text: test
 dictionary:
   one: 1
   two: 2
 number: 2
 list:
   - one
   - two
   - four
   - name: John
     age: 20
 USERNAME:
   type: env
   load: USERNAME
 callable_lambda:
   type: lambda
   load: "lambda: print('hell world!')"
 datenow:
   type: lambda
   import:
     - "from datetime import datetime as dd"
   load: "lambda: print(dd.utcnow().strftime('%Y-%m-%d'))"
 test_function:
   type: function
   name: test_function
   args:
     x: 7
     y: null
     s: hello world
     any:
       - 1
       - hello
   import:
     - "math"
   load: >
     def test(x, y, s, any):
       print(math.e)
       iii = 111
       print(iii)
       print(x,s)
       if y:
         print(type(y))
       else:
         print(y)
       for i in any:
         print(i)

Then use it as an object.

>> from pickle_rick import PickleRick

>> config = PickleRick('./config.yaml', deep=True, load_lambda=True)

>> config.BASIC.dictionary
{'one' : 1, 'two' : 2}

>> config.BASIC.callable_lambda()
hell world!

You can define Python functions, load additional data from other files or REST APIs, environmental variables, and then write everything out to YAML or JSON again.

This works especially well when building systems that require structured configuration files, or in notebooks as interactive structures.

There is a security note to using this. Only load files that are trusted, as any code can be executed, thus stay clear of just loading anything without knowing what the complete contents are.

The package is called PickleRick and is available here:

user_42434150
Related