Using the optimal learning rate results in random guessing accuracy

I am going through Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow by Aurélien Géron and I'm trying to make sense of what I'm doing wrong while solving an exercise. It's exercise 8 from Chapter 11. What I have to do is train a neural network with 20 hidden layers, 100 neurons each, with the activation function ELU and weight initializer He Normal on the CIFAR10 dataset (I know 20 hidden layers of 100 neurons is a lot, but that's the point of the exercise, so bear with me). I have to use Early Stopping and Nadam optimizer.

The problem that I have is that I didn't know what learning rate to use. In the solutions notebook, the author listed a bunch of learning rates that he tried and used the best one he found. I wasn't satisfied by this and I decided to try to find the best learning rate myself. So I used a technique that was recommended in the book: train the network for one epoch, exponentially increasing the learning rate at each iteration. Then plot the loss as a function of the learning rate, see where the loss hits its minumum, and choose a slightly smaller learning rate (since that's the upper bound).

This is the code from my model:

model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))
for _ in range(20):
    model.add(keras.layers.Dense(100,
                             activation="elu",
                             kernel_initializer="he_normal"))

model.add(keras.layers.Dense(10, activation="softmax"))
optimizer = keras.optimizers.Nadam(lr=1e-5)
model.compile(loss="sparse_categorical_crossentropy",
              optimizer=optimizer,
              metrics=["accuracy"])

(Ignore the value of the learning rate, it doesn't matter yet since I'm trying to find the right one.)

Here is the code that was used to find the optimal learning rate:

class ExponentialLearningRate(keras.callbacks.Callback):
    def __init__(self, factor):
        self.factor = factor
        self.rates = []
        self.losses = []
    def on_batch_end(self, batch, logs):
        self.rates.append(keras.backend.get_value(self.model.optimizer.lr))
        self.losses.append(logs["loss"])
        keras.backend.set_value(self.model.optimizer.lr, self.model.optimizer.lr * self.factor)

def find_learning_rate(model, X, y, epochs=1, batch_size=32, min_rate=10**-5, max_rate=10):
    init_weights = model.get_weights()
    init_lr = keras.backend.get_value(model.optimizer.lr)
    iterations = len(X) // batch_size * epochs
    factor = np.exp(np.log(max_rate / min_rate) / iterations)
    keras.backend.set_value(model.optimizer.lr, min_rate)
    exp_lr = ExponentialLearningRate(factor)
    history = model.fit(X, y, epochs = epochs, batch_size = batch_size, callbacks = [exp_lr])
    keras.backend.set_value(model.optimizer.lr, init_lr)
    model.set_weights(init_weights)
    return exp_lr.rates, exp_lr.losses

def plot_lr_vs_losses(rates, losses):
    plt.figure(figsize=(10, 5))
    plt.plot(rates, losses)
    plt.gca().set_xscale("log")
    plt.hlines(min(losses), min(rates), max(rates))
    plt.axis([min(rates), max(rates), min(losses), losses[0] + min(losses) / 2])
    plt.xlabel("Learning rate")
    plt.ylabel("Loss")

The find_learning_rate() function exponentially increases the learning rate at each iteration, going from the minimum learning rate of 10^(-5) to the maximum learning rate of 10. After that, I plotted the curve using the function plot_lr_vs_losses() and this is what I got:

Looks like using a learning rate of 1e-2 would be great, right? But when I re-compile the model, with a learning rate of 1e-2 the model's accuracy on both the training set and the validation set is about 10%, which is like choosing randomly, since we have 10 classes. I used early stopping, so I can't say that I let the model train for too many epochs (I used 100). But even during training, the model doesn't learn anything, the accuracy of both the training set and the validation set is always at around 10%.

This whole problem disappears when I use a much smaller learning rate (the one used by the author in the solutions notebook). When I use a learning rate of 5e-5 the model is learning and reaches around 50% accuracy on the validation set (which is what the exercise expects, that's the same accuracy the author got). But how is it that using the learning rate indicated by the plot is so bad? I read a little bit on the internet and this method of exponentially increasing the learning rate seems to be used by many people, so I really don't understand what I did wrong.