System information:
- Laptop PC
- OS Platform and Distribution: Ubuntu Linux, 18.04, x64
- TensorFlow installed from: pip
- TensorFlow version: 2.1.0
- Python version: 3.6.9
- GPU model and memory: nVidia RTX2060 6GB
- CPU model: i7-9850H
- RAM: 16GB
I was working with TensorFlow 2.0 on CPU with another PC.
I installed (using the guide at https://www.tensorflow.org/install/gpu ) CUDA 10.1.
I started running an old script for a NN using ResNet50V2 on a dataset of 26998 training images and 1000 as validation with 2 classes.
The NET
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
keras_layer (KerasLayer) (None, 1792) 4363712
_________________________________________________________________
dense (Dense) (None, 64) 114752
_________________________________________________________________
dropout (Dropout) (None, 64) 0
_________________________________________________________________
dense_1 (Dense) (None, 2) 130
=================================================================
Total params: 4,478,594
Trainable params: 114,882
Non-trainable params: 4,363,712
_________________________________________________________________
where keras_layer is the resnet got from tensorflow_hub.
As first issue, I got a CUDA_ERROR_OUT_OF_MEMORY that I solved adding
physical_devices = tf.config.experimental.list_physical_devices('GPU')
for dev in physical_devices:
try:
tf.config.experimental.set_memory_growth(dev, True)
print(dev, "SET MEMORY GROWTH")
except:
print("Device config error")
sys.exit(1)
however now I got a similar warning:
2020-04-07 01:39:57.857284: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 2.70G (2897281024 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-04-07 01:39:58.035192: W tensorflow/core/common_runtime/bfc_allocator.cc:309] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
both printed several times.
After this I get:
2020-04-07 01:41:59.069302: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
I read that they are not related, but it's not clear to me what may cause this second warning.
In the end, this appeared:
WARNING:tensorflow:sample_weight modes were coerced from
...
to
['...']
(I think they're caused by three different issues, I decided to post all in a single question to not spam around, but if this is a problem I can split in different threads.)
I used ImageDataGenerator to generate datasets:
train_image_generator = ImageDataGenerator(rescale=1./255., rotation_range=10., horizontal_flip=True) # Generator for our training data
validation_image_generator = ImageDataGenerator(rescale=1./255.) # Generator for our validation data
train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,
directory=train_dir,
shuffle=True,
target_size=(IMG_H, IMG_W),
class_mode='sparse')
validation_data_gen = validation_image_generator.flow_from_directory(batch_size=batch_size,
directory=validation_dir,
shuffle=True,
target_size=(IMG_H, IMG_W),
class_mode='sparse')
If some other code is needed I will add.
Thanks.
EDIT1:
for the warning:
2020-04-07 01:41:59.069302: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
I tried to set workers=1 in fit() and it disappear but I don't still know the cause and the consequences of this warning.