Pickle or json?

user_157519 0
Viewed 111097

I need to save to disk a little dict object whose keys are of the type str and values are ints and then recover it. Something like this:

{'juanjo': 2, 'pedro':99, 'other': 333}

What is the best option and why? Serialize it with pickle or with simplejson?

I am using Python 2.6.

pickle python json
8 Answers

I prefer JSON over pickle for my serialization. Unpickling can run arbitrary code, and using pickle to transfer data between programs or store data between sessions is a security hole. JSON does not introduce a security hole and is standardized, so the data can be accessed by programs in different languages if you ever need to.

user_1928390

If you do not have any interoperability requirements (e.g. you are just going to use the data with Python) and a binary format is fine, go with cPickle which gives you really fast Python object serialization.

If you want interoperability or you want a text format to store your data, go with JSON (or some other appropriate format depending on your constraints).

user_942370

JSON or pickle? How about JSON and pickle!

You can use jsonpickle. It easy to use and the file on disk is readable because it's JSON.

See jsonpickle Documentation

user_998960

Most answers are quite old and miss some info.

For the statement "Unpickling can run arbitrary code":
  1. Check the example in https://docs.python.org/3/library/pickle.html#restricting-globals
import pickle
pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
pickle.loads(b"cos\nsystem\n(S'pwd'\ntR.")

pwd can be replaced e.g. by rm to delete files.

  1. Check https://checkoway.net/musings/pickle/ for more sophisicated "run arbitrary code" template. The code is written in python2.7 but I guess with some modification, could also work in python3. If you make it work in python3, please add the python3 version my answer. :)
For the "pickle speed vs json" part:

Firstly, there is no explicit cpickle in python3 now .

And for this test code borrowed from another answer, pickle beats json in all:

import pickle
import json, random
from time import time
from hashlib import md5

test_runs = 100000

if __name__ == "__main__":
    payload = {
        "float": [(random.randrange(0, 99) + random.random()) for i in range(1000)],
        "int": [random.randrange(0, 9999) for i in range(1000)],
        "str": [md5(str(random.random()).encode('utf8')).hexdigest() for i in range(1000)]
    }
    modules = [json, pickle]

    for payload_type in payload:
        data = payload[payload_type]
        for module in modules:
            start = time()
            if module.__name__ in ['pickle']:
                for i in range(test_runs): serialized = module.dumps(data)
            else:
                for i in range(test_runs): 
                    # print(i)
                    serialized = module.dumps(data)
            w = time() - start
            start = time()
            for i in range(test_runs):
                unserialized = module.loads(serialized)
            r = time() - start
            print("%s %s W %.3f R %.3f" % (module.__name__, payload_type, w, r))

result:

tian@tian-B250M-Wind:~/playground/pickle_vs_json$ p3 pickle_test.py 
json float W 41.775 R 26.738
pickle float W 1.272 R 2.286
json int W 5.142 R 4.974
pickle int W 0.589 R 1.352
json str W 10.379 R 4.626
pickle str W 3.062 R 3.294
user_59838410
Related