Convert Dictionary To Key and Value Tensor

Viewed 241

I am dealing with a large scale dictionary (2M+ key,value pairs). I want to keep the information stored in memory, not having to resort to writing to disk. I tested with other formats besides dictionaries including Pytorch Tensors and it seems that would be more efficient. I have included the code from my experiments below:

import torch
import sys
import random
source = list(random.choices(list(range(0,10)), k=100))
dest = list(range(100))
ls1 = torch.Tensor([source, dest])
dic1 = {}
for i in range(len(source)):
    src = source[i]
    dst = dest[i]
    if src not in dic1:
        dic1[src] = []
    dic1[src].append(dst)

sys.getsizeof(ls1)
# >>> 64
sys.getsizeof(dic1)
# >>> 364

Based on this experiment, if storing into a PyTorch tensor is the most memory efficient, what would be the fastest way to convert from a dictionary to a tensor? My use case dictates using a loop that calls a function that returns a large dictionary. While a single dictionary returned by the function fits in memory, combining it with another dictionary would cause a memory issue. My solution is at every iteration of the loop to take the dictionary, convert it into a PyTorch tensor, and combine it with the previous iterations' tensor.

0 Answers
Related