for some reason I have to save data without other python library such as pandas and numpy, then I need to load data in tensorflow. here is the data format and code example.
Note that the length of 'xref' and 'asm' is uncertain and this triplet is just one sample in my dataset
file_paths = ['1.csv']
ea = 1 # int32
xref = [1,2] # N-list int32
asm = [[4,5,6,7,8],[9,0,1,2,3]] # Mx5-list float32
with open('1.csv','w') as f:
csv_f = csv.writer(f)
csv_f.writerow((ea,xref,asm))
# CSV file:
# 1,"[1, 2]","[[4, 5, 6, 7, 8], [9, 0, 1, 2, 3]]"
#loading
dataset = tf.data.TextLineDataset(file_paths)
for textline in dataset:
record_default = [0,'0','0']
ea,xref,asm = tf.io.decode_csv(textline,record_defaults=record_default)
the data loaded as follow:
<tf.Tensor: shape=(), dtype=int32, numpy=1>
<tf.Tensor: shape=(), dtype=string, numpy=b'[1, 2]'>
<tf.Tensor: shape=(), dtype=string, numpy=b'[[4, 5, 6, 7, 8], [9, 0, 1, 2, 3]]'>
it still far from my target which data should be loaded like
<tf.Tensor: shape=(), dtype=int32, numpy=1>
<tf.Tensor: shape=(2,), dtype=int32, numpy=array([1, 2])>
<tf.Tensor: shape=(2, 5), dtype=float32, numpy= array([[4, 5, 6, 7, 8], [9, 0, 1, 2, 3]])>
So what can I do? Actually I want to use dataset.map(func),I tried json format,it works but is too slow for tensorflow.