I have a CNN training scheme where I defined methods to import and preprocess data using the tf.data.Dataset API and eventually training with keras fit.
my model trains on large arrays of 5 dim where each array is stored in a .mat file and usign the below imported and preprocessed. I define a train, validation and test dataset within a class that I only use to initialize the dataset itself. the dataset structure:
output_types = dict()
for key, val in self.list_IDs[0].items():
output_types[key] = tf.convert_to_tensor(val).dtype
gen = data_gen(self.list_IDs, self.shuffle) ##class that has generator as method
ds = tf.data.Dataset.from_generator(generator=gen.file_and_config_gen, output_types=output_types)
ds = ds.map(load_data, num_parallel_calls=self.num_parallel_calls)
ds = ds.batch(self.batch_size, drop_remainder=False)
ds = ds.repeat(self.epochs)
in the above all the attributes are for the class that the dataset defined within. the generator defined the following way within a class:
class data_gen:
def __init__(self, file_list, shuffle):
self.file_list = file_list
self.shuffle = shuffle
def file_and_config_gen(self):
print('starting generator and shuffle list')
self.indexes = np.arange(len(self.file_list))
if self.shuffle:
np.random.shuffle(self.indexes)
for i in self.indexes:
ID = self.file_list[int(i)]
yield ID
the list named file_list and list_IDs contain list of dictionaries where each dictionary has import settings and preprocess for a specific file.
the map function defined bellow:
def load_data(self, config_dict):
x, y = tf.py_function(_read_mat, [config_dict['file_path'], self.x_field, self.y_field], (tf.complex128, tf.float64))
x = tf.reshape(x, self.initial_shape_import)
y = tf.reshape(y, self.y_shape_import)
if self.aug_en:
begin = [0] * len(x.shape.as_list())
begin[self.x_aug_ch] = config_dict['x_aug_slice'][0]
size = x.shape.as_list()
size_aug_dim = self.dim_x[self.x_aug_ch]
size[self.x_aug_ch] = size_aug_dim
x = tf.slice(x, begin, size)
begin = [0] * len(y.shape.as_list())
begin[self.y_aug_ch] = config_dict['y_aug_slice'][0]
size = y.shape.as_list()
size_aug_dim = self.dim_y[self.y_aug_ch]
size[self.y_aug_ch] = size_aug_dim
y = tf.slice(y, begin, size)
if self.make_y_on_x_grid:
y = tf.image.resize(y, [self.lateral_grid, self.lateral_grid], method=tf.image.ResizeMethod.NEAREST_NEIGHBOR,
preserve_aspect_ratio=True)
y.set_shape(self.dim_y)
x.set_shape(self.dim_x)
if self.x_type == 'abs':
x = tf.math.abs(x)
if self.standardize_norm_data:
x = (x - self.mean_data[0]) / tf.sqrt(self.std_data[0])
return x, y
else:
real_x, imag_x = tf.math.real(x), tf.math.imag(x)
if self.standardize_norm_data:
real_x = (real_x - self.mean_data[0]) / tf.sqrt(self.std_data[0])
imag_x = (imag_x - self.mean_data[1]) / tf.sqrt(self.std_data[1])
return (real_x, imag_x), y
where the read from .mat file is actually:
from scipy.io import loadmat
def _read_mat(filepath, x_field_name, y_field_name):
f = loadmat(filepath.numpy().decode('ASCII'), variable_names=[x_field_name.numpy().decode('ASCII'), y_field_name.numpy().decode('ASCII')])
return f[x_field_name.numpy().decode('ASCII')], f[y_field_name.numpy().decode('ASCII')]
I use the loading in parralel using the map function since I got lost with interleave. I train calling the keras.fit:
model.fit(x=x, y=y, validation_data=validation_data, epochs=epochs, steps_per_epoch=steps_per_epoch, workers=workers,
validation_steps=validation_steps, validation_split=validation_split, batch_size=batch_size)
where x, validation_data are the above specified dataset but using different list_IDs for each. (eventually each object in the list is a single sample).
I run the above and starting from the last batch (I defined the steps to be smaller to exclude last incomplete batch) I get errors of running out of memory.
the main issues: 1. The memory loads entirely and the program stops.
2. the program also enters to a deadlock where the training stucks when it has enough memory
3. the generator called from the beginning several times when the repeats and the steps of
the generator fit the epoch and batch counts.
bellow is the printing to the cmd line from where it starts training. I added printing of the dataset objects and training test sizes. I use tf 2.1, python 3.7 on either windows 10 and unix. I use nvidia geforce gtx 1080 Ti and on another machine I use DGX station with Tesla V100.
Train for 74 steps, validate for 7 steps
Epoch 1/10
starting generator and shuffle list
2020-05-06 23:02:10.306681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-06 23:02:10.617387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-06 23:02:11.837448: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation. This message will be only logged once.
73/74 [============================>.] - ETA: 6s - loss: 0.0068 - accuracy: 0.8548 - mse: 0.0068 starting generator and shuffle list
starting generator and shuffle list
2020-05-06 23:11:39.180994: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 17179869184 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:11:39.187845: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 17179869184
2020-05-06 23:11:39.319512: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 15461881856 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:11:39.324774: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 15461881856
2020-05-06 23:11:39.440502: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 13915693056 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:11:39.444384: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 13915693056
74/74 [==============================] - 595s 8s/step - loss: 0.0068 - accuracy: 0.8550 - mse: 0.0068 - val_loss: 0.0046 - val_accuracy: 0.8561 - val_mse: 0.0046
Epoch 2/10
6/74 [=>............................] - ETA: 1:59 - loss: 0.0056 - accuracy: 0.8549 - mse: 0.0056starting generator and shuffle list
16/74 [=====>........................] - ETA: 6:59 - loss: 0.0056 - accuracy: 0.8533 - mse: 0.0056starting generator and shuffle list
20/74 [=======>......................] - ETA: 7:32 - loss: 0.0054 - accuracy: 0.8540 - mse: 0.00542020-05-06 23:14:57.484569: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:57.488560: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:57.806151: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:57.810867: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:58.171997: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:58.175640: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:58.556121: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:58.561653: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:58.934121: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:58.938117: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:59.221201: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:59.226916: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:14:59.525388: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:14:59.530471: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
21/74 [=======>......................] - ETA: 7:42 - loss: 0.0054 - accuracy: 0.8540 - mse: 0.00542020-05-06 23:15:08.166446: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:08.170612: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:08.473786: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:08.477095: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:08.723715: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:08.727838: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:08.978101: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:08.985016: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:09.224867: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:09.229381: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
22/74 [=======>......................] - ETA: 7:38 - loss: 0.0053 - accuracy: 0.8541 - mse: 0.00532020-05-06 23:15:16.810263: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:16.814654: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:17.068734: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:17.075101: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:27.351469: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:27.359544: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:27.639494: E tensorflow/stream_executor/cuda/cuda_driver.cc:863] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-05-06 23:15:27.643745: W .\tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 34359738368
2020-05-06 23:15:27.646560: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (gpu_host_bfc) ran out of memory trying to allocate 889.89MiB (rounded to 933120000). Current allocation summary follows.
2020-05-06 23:15:27.653039: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for gpu_host_bfc
2020-05-06 23:15:27.655249: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256): Total Chunks: 18, Chunks in use: 18. 4.5KiB allocated for chunks. 4.5KiB in use in bin. 188B client-requested in use in bin.
2020-05-06 23:15:27.659356: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.663243: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.667674: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.671611: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.675500: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.679394: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.685740: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.689666: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.694015: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.699658: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.704774: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288): Total Chunks: 1, Chunks in use: 0. 1019.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.709846: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.718097: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.723683: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.729806: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608): Total Chunks: 16, Chunks in use: 15. 180.32MiB allocated for chunks. 169.34MiB in use in bin. 164.79MiB client-requested in use in bin.
2020-05-06 23:15:27.735560: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.739513: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.743982: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.752531: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 182.97MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-05-06 23:15:27.758239: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456): Total Chunks: 33, Chunks in use: 30. 27.31GiB allocated for chunks. 26.16GiB in use in bin. 26.07GiB client-requested in use in bin.
2020-05-06 23:15:27.764473: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 889.89MiB was 256.00MiB, Chunk State:
2020-05-06 23:15:27.768356: I tensorflow/core/common_runtime/bfc_allocator.cc:923] Size: 268.21MiB | Requested Size: 16B | in_use: 0 | bin_num: 20, prev: Size: 889.89MiB | Requested Size: 889.89MiB | in_use: 1 | bin_num: -1
2020-05-06 23:15:27.772774: I tensorflow/core/common_runtime/bfc_allocator.cc:923] Size: 375.33MiB | Requested Size: 4B | in_use: 0 | bin_num: 20, prev: Size: 889.89MiB | Requested Size: 889.89MiB | in_use: 1 | bin_num: -1
2020-05-06 23:15:27.781140: I tensorflow/core/common_runtime/bfc_allocator.cc:923] Size: 536.43MiB | Requested Size: 16B | in_use: 0 | bin_num: 20, prev: Size: 889.89MiB | Requested Size: 889.89MiB | in_use: 1 | bin_num: -1
2020-05-06 23:15:27.785531: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1048576
2020-05-06 23:15:27.787679: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400000 of size 256 next 1
2020-05-06 23:15:27.790016: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400100 of size 256 next 2
2020-05-06 23:15:27.793309: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400200 of size 256 next 3
2020-05-06 23:15:27.796157: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400300 of size 256 next 4
2020-05-06 23:15:27.799000: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400400 of size 256 next 5
2020-05-06 23:15:27.802126: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400500 of size 256 next 6
2020-05-06 23:15:27.804496: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400600 of size 256 next 7
2020-05-06 23:15:27.810142: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400700 of size 256 next 8
2020-05-06 23:15:27.812679: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400800 of size 256 next 9
2020-05-06 23:15:27.815004: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400900 of size 256 next 10
2020-05-06 23:15:27.817357: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400a00 of size 256 next 11
2020-05-06 23:15:27.819784: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400b00 of size 256 next 12
2020-05-06 23:15:27.822918: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400c00 of size 256 next 13
2020-05-06 23:15:27.826392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400d00 of size 256 next 14
2020-05-06 23:15:27.829365: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400e00 of size 256 next 26
2020-05-06 23:15:27.831838: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203400f00 of size 256 next 24
2020-05-06 23:15:27.834410: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203401000 of size 256 next 23
2020-05-06 23:15:27.836764: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203401100 of size 256 next 18
2020-05-06 23:15:27.842107: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 203401200 of size 1043968 next 18446744073709551615
2020-05-06 23:15:27.844875: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1073741824
2020-05-06 23:15:27.847025: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 203600000 of size 1025280000 next 44
2020-05-06 23:15:27.849456: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2407c8800 of size 11520000 next 47
2020-05-06 23:15:27.852011: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2412c5000 of size 11520000 next 51
2020-05-06 23:15:27.855183: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 241dc1800 of size 11520000 next 54
2020-05-06 23:15:27.857892: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2428be000 of size 13901824 next 18446744073709551615
2020-05-06 23:15:27.860911: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1073741824
2020-05-06 23:15:27.863559: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 243600000 of size 933120000 next 43
2020-05-06 23:15:27.866005: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27afe4800 of size 11520000 next 33
2020-05-06 23:15:27.868392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27bae1000 of size 11520000 next 35
2020-05-06 23:15:27.874496: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27c5dd800 of size 11520000 next 39
2020-05-06 23:15:27.877340: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27d0da000 of size 11520000 next 32
2020-05-06 23:15:27.879780: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27dbd6800 of size 11520000 next 25
2020-05-06 23:15:27.882212: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27e6d3000 of size 11520000 next 55
2020-05-06 23:15:27.885197: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27f1cf800 of size 11520000 next 56
2020-05-06 23:15:27.887578: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 27fccc000 of size 11520000 next 63
2020-05-06 23:15:27.889964: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2807c8800 of size 11520000 next 64
2020-05-06 23:15:27.892345: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2812c5000 of size 11520000 next 67
2020-05-06 23:15:27.894728: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 281dc1800 of size 11520000 next 70
2020-05-06 23:15:27.897160: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2828be000 of size 13901824 next 18446744073709551615
2020-05-06 23:15:27.901880: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 2147483648
2020-05-06 23:15:27.907589: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 283600000 of size 933120000 next 52
2020-05-06 23:15:27.910125: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 2bafe4800 of size 933120000 next 22
2020-05-06 23:15:27.912860: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 2f29c9000 of size 281243648 next 18446744073709551615
2020-05-06 23:15:27.915652: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 4294967296
2020-05-06 23:15:27.918425: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 303a00000 of size 933120000 next 48
2020-05-06 23:15:27.920888: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 33b3e4800 of size 933120000 next 40
2020-05-06 23:15:27.923344: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 372dc9000 of size 933120000 next 30
2020-05-06 23:15:27.925741: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 3aa7ad800 of size 933120000 next 36
2020-05-06 23:15:27.928094: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 3e2192000 of size 562487296 next 18446744073709551615
2020-05-06 23:15:27.930778: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 8589934592
2020-05-06 23:15:27.936683: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 403a00000 of size 933120000 next 41
2020-05-06 23:15:27.939106: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 43b3e4800 of size 933120000 next 45
2020-05-06 23:15:27.941529: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 472dc9000 of size 933120000 next 34
2020-05-06 23:15:27.943952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 4aa7ad800 of size 933120000 next 38
2020-05-06 23:15:27.946331: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 4e2192000 of size 933120000 next 42
2020-05-06 23:15:27.949448: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 519b76800 of size 933120000 next 31
2020-05-06 23:15:27.952226: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 55155b000 of size 933120000 next 19
2020-05-06 23:15:27.954579: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 588f3f800 of size 933120000 next 21
2020-05-06 23:15:27.956960: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 5c0924000 of size 933120000 next 27
2020-05-06 23:15:27.959313: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 5f8308800 of size 191854592 next 18446744073709551615
2020-05-06 23:15:27.961996: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 12524123136
2020-05-06 23:15:27.967098: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 603a00000 of size 933120000 next 50
2020-05-06 23:15:27.969450: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 63b3e4800 of size 933120000 next 53
2020-05-06 23:15:27.971800: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 672dc9000 of size 933120000 next 46
2020-05-06 23:15:27.974361: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 6aa7ad800 of size 933120000 next 58
2020-05-06 23:15:27.976726: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 6e2192000 of size 933120000 next 16
2020-05-06 23:15:27.980288: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 719b76800 of size 933120000 next 57
2020-05-06 23:15:27.983390: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 75155b000 of size 933120000 next 29
2020-05-06 23:15:27.986713: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 788f3f800 of size 933120000 next 60
2020-05-06 23:15:27.989122: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7c0924000 of size 933120000 next 62
2020-05-06 23:15:27.991528: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 7f8308800 of size 933120000 next 65
2020-05-06 23:15:27.995357: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 82fced000 of size 933120000 next 61
2020-05-06 23:15:28.000221: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 8676d1800 of size 933120000 next 68
2020-05-06 23:15:28.003281: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at 89f0b6000 of size 933120000 next 66
2020-05-06 23:15:28.005785: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at 8d6a9a800 of size 393563136 next 18446744073709551615
2020-05-06 23:15:28.008652: I tensorflow/core/common_runtime/bfc_allocator.cc:955] Summary of in-use Chunks by size:
2020-05-06 23:15:28.011532: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 18 Chunks of size 256 totalling 4.5KiB
2020-05-06 23:15:28.013833: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 13 Chunks of size
this is where these line repeat and the program crashes