I have this training set of about 1.4 GB, but I can't load it in a TabularDataset.
>>> REF = data.Field(lower=True, tokenize=tokenize_char, init_token='<sos>',eos_token='<eos>')
>>> SRC = data.Field(lower=True, tokenize=tokenize_char)
>>> train = data.TabularDataset('./train.csv', format='csv', fields= [('src', SRC), ('ref', REF)])
Killed
For small dataset, this code works fine. I don't really understand why the process got killed, since the dataset is not enormous large. Any idea how I can load this file? thanks!