How are accuracies of runs with different shuffles distributed?

user_4398524 0
Viewed 42

I set up a network to learn Fashion MNIST in the style of Hands On Machine Learning, page 298. When I ran the code multiple times, the accuracy was slightly different each time. This made me wonder if the accuracies were normally distributed around some mean, and that with enough runs, I could accurately determine the population mean of all runs for those parameters.

So I ran this code, which fits the model 1000 times, each with differently shuffled training data:

from tensorflow.keras import datasets, models, layers
import matplotlib.pyplot as plt

(X_train, y_train), _ = datasets.fashion_mnist.load_data()
final_accuracies = []
for i in range(1000):
    model = models.Sequential(layers=[
        layers.Flatten(input_shape=[28, 28]),
        layers.Dense(300, activation="relu"),
        layers.Dense(100, activation="relu"),
        layers.Dense(10, activation="softmax")])
    model.compile(loss="sparse_categorical_crossentropy", optimizer="sgd", metrics=["accuracy"])
    history = model.fit(X_train / 255.0, y_train, epochs=5, validation_split=0.2)
    final_accuracies.append(history.history['val_accuracy'][-1])

plt.hist(final_accuracies, bins=30)
plt.show()

The resulting distribution of accuracies is shown in the attached histogram.

Accuracy histogram

What kind of distribution is this?! It's clearly not normally distributed. It's got a much longer tail in the direction of lower accuracies.

Is the statistics of what kind of distribution these accuracies are drawn from worked out? If so, please illuminate me.

Also, if this question really fits in better at stats.stackexchange.com, please kindly let me know that all I'll remove it from here and post it there instead.

mnist keras tensorflow statistics python
1 Answers

I do not know the answer to your question specifically. It would be difficult to figure out because there are numerous sources of randomness at play each time you run a model. One example is the weight initialization. Others can be due to activities which shuffle the data. Dropout layers etc. If you use transfer learning randomness maybe be included within the model you select. It would take a lot of work to combine all these sources to mathematically calculate the distribution. There are a lot of question on Stack Overflow on how to get repeatable results in tensorflow. Turns out to do so you have to hunt down every source of random processes and try to eliminate it by using a "seed" that makes the random activity repeatable for each time the model is run. This is no easy task.

user_107989170
Related