Saving the dictionary with str keys and a list of pytorch tensors as values

Viewed 2164

I want to save a list of dictionary in which keys are indices of queries (so the keys of the dictionary are strings). The values of this dictionary are list of sparse tensors which I store as follows :

{'1185869': [tensor(indices=tensor([[   31,    59,   528,  7227, 14981]]),
         values=tensor([1., 1., 1., 1., 1.]),
         size=(30000,), nnz=5, layout=torch.sparse_coo),
  tensor(indices=tensor([[ 39,  74, 233]]),
         values=tensor([1., 1., 1.]),
         size=(30000,), nnz=3, layout=torch.sparse_coo),
  tensor(indices=tensor([[0, 1, 2]]),
         values=tensor([1., 1., 1.]),
         size=(30000,), nnz=3, layout=torch.sparse_coo),
  tensor(indices=tensor([[  36,   99,  243,  428,  513,  514,  741,  751, 1615]]),
         values=tensor([1., 1., 1., 1., 1., 1., 1., 1., 1.]),
         size=(30000,), nnz=9, layout=torch.sparse_coo),
  tensor(indices=tensor([[ 170,  356,  513,  615,  771, 1117]]),
         values=tensor([1., 1., 1., 1., 1., 1.]),
         size=(30000,), nnz=6, layout=torch.sparse_coo),
  tensor(indices=tensor([[ 9, 11]]),
         values=tensor([1., 1.]),
         size=(30000,), nnz=2, layout=torch.sparse_coo)]}

I've not saved any such dictionary before. Generally for normal dictionaries, we pickle it. If I pickle this dictionary, I'm getting a pytorch warning. This makes me doubt the method I'm using to save the dictionary. What extensions should I use to save such dictionary? Is pickling such dictionary safe? What method should I use to save it? Thanks in advance.

1 Answers

If you are talking about this warning:

/install/lib/python3.6/site-packages/torch/storage.py:34: FutureWarning: pickle support for Storage will be removed in 1.5. Use `torch.save` instead
  warnings.warn("pickle support for Storage will be removed in 1.5. Use `torch.save` instead", FutureWarning)

Then, good for you, it is not going to be deprecated actually. See this PR. Pytorch 1.6.0 just released yesterday, and indeed there is no more warning. So update Pytorch to get rid of it!

Related