I spent hours configuring my computer and finally got to make python train data on GPUs instead of CPU. However, for some reason my models keep getting interrupted halfway during the middle of their epochs and I cannot complete the training of the models.
Waiting for the computer does not solve this problem and I cannot interrupt the kernel either. I tried other people's solution and still don't have much luck.
I can train my model normally if I'm using a CPU (at a crawling rate), but when I switch to GPU my model trains really fast before they hang up half way, not completing all the epochs that are required. My python kernel also gets stuck on running after that and I cannot interrupt it unless I terminate the whole thing from task manager.
From my task manager performance history, during training there is a sustained spike at my GPU, which is expected. But when it hangs up my GPU activity goes back to 0, even though my kernel indicates that the training is still in the middle of its epoch. This happens randomly and is not dependent on the timing or the number of epochs, although it is more likely to happen the longer I train the data.
Here is my sequential model.
def prepare_sequences(notes, n_vocab, seq_len):
""" Prepare the sequences used by the Neural Network """
sequence_length = seq_len
names = sorted(set(item for item in notes))
note_to_int = dict((note, number) for number, note in enumerate(names))
network_input = []
network_output = []
# create input sequences and the corresponding outputs
for i in range(0, len(notes) - sequence_length, 1):
sequence_in = notes[i:i + sequence_length]
sequence_out = notes[i + sequence_length]
network_input.append([note_to_int[char] for char in sequence_in])
network_output.append(note_to_int[sequence_out])
n_patterns = len(network_input)
# reshape the input into a format compatible with LSTM layers
network_input = numpy.reshape(network_input, (n_patterns, sequence_length, 1))
# normalize input
network_input = network_input / float(n_vocab)
network_output = np_utils.to_categorical(network_output)
return (network_input, network_output)
def create_network(network_input, n_vocab, LSTM_node_count, Dropout_count):
""" create the structure of the neural network """
model = Sequential()
model.add(LSTM(
LSTM_node_count,
input_shape=(network_input.shape[1], network_input.shape[2]),
recurrent_dropout= Dropout_count,
return_sequences=True
))
model.add(LSTM(
LSTM_node_count,
return_sequences=True,
recurrent_dropout= Dropout_count,))
model.add(LSTM(LSTM_node_count))
model.add(BatchNorm())
model.add(Dropout(Dropout_count))
model.add(Dense(256))
model.add(Activation('relu'))
model.add(BatchNorm())
model.add(Dropout(Dropout_count))
model.add(Dense(n_vocab))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
return model
def train(model, network_input, network_output, epoch, batchsize):
""" train the neural network """
filepath = "trained_weights/" + "weights-improvement-{epoch:02d}-{loss:.4f}-bigger.hdf5"
checkpoint = ModelCheckpoint(
filepath,
monitor='loss',
verbose=0,
save_best_only= True,
mode='min'
)
callbacks_list = [checkpoint]
model.fit(network_input,
network_output,
epochs= epoch,
batch_size= batchsize,
callbacks= callbacks_list)
configproto = tf.compat.v1.ConfigProto()
configproto.gpu_options.allow_growth = True
configproto.gpu_options.polling_inactive_delay_msecs = 10
sess = tf.compat.v1.Session(config=configproto)
tf.compat.v1.keras.backend.set_session(sess)
During training, I also get a warning message and I don't know what it means.
WARNING:tensorflow:Layer lstm will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
WARNING:tensorflow:Layer lstm_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU
C:\Users\David>nvidia-smi
Sun Dec 27 15:56:16 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.89 Driver Version: 460.89 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1050 WDDM | 00000000:01:00.0 Off | N/A |
| N/A 47C P8 N/A / N/A | 120MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 5496 C+G ...5n1h2txyewy\SearchApp.exe N/A |
| 0 N/A N/A 7372 C+G ...nputApp\TextInputHost.exe N/A |
| 0 N/A N/A 8268 C+G ...wekyb3d8bbwe\Music.UI.exe N/A |
| 0 N/A N/A 9420 C+G ...artMenuExperienceHost.exe N/A |
| 0 N/A N/A 10084 C+G ...ekyb3d8bbwe\YourPhone.exe N/A |
| 0 N/A N/A 11292 C+G Insufficient Permissions N/A |
| 0 N/A N/A 14684 C+G ...cw5n1h2txyewy\LockApp.exe N/A |
+-----------------------------------------------------------------------------+
I am currently using tensorflow 2.4, CUDA 11.2,