How can I execute multiple NN training?

Question

I have two NVidia GPUs in the machine, but I am not using them.

I have three NN training running on my machine. When I am trying to run the fourth one, the script is giving me the following error:

my_user@my_machine:~/my_project/training_my_project$ python3 my_project.py
Traceback (most recent call last):
  File "my_project.py", line 211, in 
    load_data(
  File "my_project.py", line 132, in load_data
    tx = tf.convert_to_tensor(data_x, dtype=tf.float32)
  File "/home/my_user/.local/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/my_user/.local/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 106, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Failed to allocate scratch buffer for device 0
my_user@my_machine:~/my_project/training_my_project$

How can I resolve this issue?

The following is my RAM usage:

my_user@my_machine:~/my_project/training_my_project$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15947        6651        3650          20        5645        8952
Swap:          2047         338        1709
my_user@my_machine:~/my_project/training_my_project$

The following is my CPU usage:

my_user@my_machine:~$ top -i
top - 12:46:12 up 79 days, 21:14,  2 users,  load average: 4,05, 3,82, 3,80
Tasks: 585 total,   2 running, 583 sleeping,   0 stopped,   0 zombie
%Cpu(s): 11,7 us,  1,6 sy,  0,0 ni, 86,6 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :  15947,7 total,   3638,3 free,   6662,7 used,   5646,7 buff/cache
MiB Swap:   2048,0 total,   1709,4 free,    338,6 used.   8941,6 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2081821 my_user  20   0   48,9g   2,5g 471076 S 156,1  15,8   1832:54 python3
2082196 my_user  20   0   48,8g   2,6g 467708 S 148,5  16,8   1798:51 python3
2076942 my_user  20   0   47,8g   1,6g 466916 R 147,5  10,3   2797:51 python3
   1594 gdm       20   0 3989336  65816  31120 S   0,7   0,4  38:03.14 gnome-shell
     93 root      rt   0       0      0      0 S   0,3   0,0   0:38.42 migration/13
   1185 root     -51   0       0      0      0 S   0,3   0,0   3925:59 irq/54-nvidia
2075861 root      20   0       0      0      0 I   0,3   0,0   1:30.17 kworker/22:0-events
2076418 root      20   0       0      0      0 I   0,3   0,0   1:38.65 kworker/1:0-events
2085325 root      20   0       0      0      0 I   0,3   0,0   1:17.15 kworker/3:1-events
2093002 root      20   0       0      0      0 I   0,3   0,0   1:00.05 kworker/23:0-events
2100000 root      20   0       0      0      0 I   0,3   0,0   0:45.78 kworker/2:2-events
2104688 root      20   0       0      0      0 I   0,3   0,0   0:33.08 kworker/9:0-events
2106767 root      20   0       0      0      0 I   0,3   0,0   0:25.16 kworker/20:0-events
2115469 root      20   0       0      0      0 I   0,3   0,0   0:01.98 kworker/11:2-events
2115470 root      20   0       0      0      0 I   0,3   0,0   0:01.96 kworker/12:2-events
2115477 root      20   0       0      0      0 I   0,3   0,0   0:01.95 kworker/30:1-events
2116059 my_user  20   0   23560   4508   3420 R   0,3   0,0   0:00.80 top

The following is my TF configuration:

import os

os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
# os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
# os.environ["CUDA_VISIBLE_DEVICES"] = "99" # Use both gpus for training.


import sys, random
import time
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.callbacks import ModelCheckpoint
import numpy as np
from lxml import etree, objectify


# 
# resolve GPU related issues.
try:
    physical_devices = tf.config.list_physical_devices('GPU') 
    for gpu_instance in physical_devices: 
        tf.config.experimental.set_memory_growth(gpu_instance, True)
except Exception as e:
    pass
# END of try
#

Please, take the commented lines as commented-out lines.

Relevant source code:

def load_data(fname: str, class_index: int, feature_start_index: int, **selection):
    i = 0
    file = open(fname)
    if "top_n_lines" in selection:
        lines = [next(file) for _ in range(int(selection["top_n_lines"]))]
    elif "random_n_lines" in selection:
        tmp_lines = file.readlines()
        lines = random.sample(tmp_lines, int(selection["random_n_lines"]))
    else:
        lines = file.readlines()

    data_x, data_y = [], []
    for l in lines:
        row = l.strip().split()
        x = [float(ix) for ix in row[feature_start_index:]]
        y = encode(row[class_index])
        data_x.append(x)
        data_y.append(y)  
    # END for l in lines

    num_rows = len(data_x)
    given_fraction = selection.get("validation_part", 1.0)
    if given_fraction > 0.9999:
        valid_x, valid_y = data_x, data_y
    else:
        n = int(num_rows * given_fraction)
        data_x, data_y = data_x[n:], data_y[n:]
        valid_x, valid_y = data_x[:n], data_y[:n]
    # END of if-else block

    tx = tf.convert_to_tensor(data_x, np.float32)
    ty = tf.convert_to_tensor(data_y, np.float32)
    
    vx = tf.convert_to_tensor(valid_x, np.float32)
    vy = tf.convert_to_tensor(valid_y, np.float32)  

    return tx, ty, vx, vy
# END of the function

user_13526512 · Answer

Using multiple GPUs

If developing on a system with a single GPU, you can simulate multiple GPUs with virtual devices. This enables easy testing of multi-GPU setups without requiring additional resources.

gpus = tf.config.list_physical_devices('GPU')
if gpus:
  # Create 2 virtual GPUs with 1GB memory each
  try:
    tf.config.set_logical_device_configuration(
        gpus[0],
        [tf.config.LogicalDeviceConfiguration(memory_limit=1024),
         tf.config.LogicalDeviceConfiguration(memory_limit=1024)])
    logical_gpus = tf.config.list_logical_devices('GPU')
    print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Virtual devices must be set before GPUs have been initialized
    print(e)

NOTE: Virtual devices cannot be modified after being initialized

Once there are multiple logical GPUs available to the runtime, you can utilize the multiple GPUs with tf.distribute.Strategy or with manual placement.

With tf.distribute.Strategy best practice for using multiple GPUs, here is a simple example:

tf.debugging.set_log_device_placement(True)
gpus = tf.config.list_logical_devices('GPU')
strategy = tf.distribute.MirroredStrategy(gpus)
with strategy.scope():
  inputs = tf.keras.layers.Input(shape=(1,))
  predictions = tf.keras.layers.Dense(1)(inputs)
  model = tf.keras.models.Model(inputs=inputs, outputs=predictions)
  model.compile(loss='mse',
                optimizer=tf.keras.optimizers.SGD(learning_rate=0.2))

This program will run a copy of your model on each GPU, splitting the input data between them, also known as "data parallelism".

For more information about distribution strategies or manual placement, check out the guides on the links.

user_10648765 · Answer

The RAM complaint isn't about your system ram (call it CPU RAM). It's about your GPU RAM.

The moment TF loads, it allocates all the GPU RAM for itself (some small fraction is left over due to page size stuff).

Your sample makes TF dynamically allocate GPU RAM, but it could still end up using up all the GPU RAM. Use the code below to provide a hard stop on GPU RAM per process. you'll likely want to change 1024 to 8096 or something like that.

and FYI, use nvidia-smi to monitor your GPU ram usage.

From the docs:

https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth

gpus = tf.config.list_physical_devices('GPU')
if gpus:
  # Restrict TensorFlow to only allocate 1GB of memory on the first GPU
  try:
    tf.config.set_logical_device_configuration(
        gpus[0],
        [tf.config.LogicalDeviceConfiguration(memory_limit=1024)])
    logical_gpus = tf.config.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Virtual devices must be set before GPUs have been initialized
    print(e)

How can I execute multiple NN training?

2 Answers