How to implement hyperparameter search in Tensorflow?

You can use DyTB (dynamic training bench): this tool allows you to focus only on the hyperparameter search, using tensorboard to compare the measured stats of the varisous trained model.

DyTB creates for you a unique name associated with the current set of hyperparameters and use it as a log dir. Creating different log directories allows the use of Tensorboard for an easy comparison.

For instance, you can train VGG on Cifar10 with this single line (both VGG and Cifar10 are some of predefined model & dataset available):

import tensorflow as tf
from dytb.inputs.predefined import Cifar10
from dytb.train import train
from dytb.models.predefined.VGG import VGG

# Instantiate the model
vgg = VGG()

# Instantiate the CIFAR-10 input source
cifar10 = Cifar10.Cifar10()

# 1: Train VGG on Cifar10 for 50 epochs
# Place the train process on GPU:0
device = '/gpu:0'
with tf.device(device):
    info = train(
        model=vgg,
        dataset=cifar10,
        hyperparameters={
            "epochs": 50,
            "batch_size": 50,
            "regularizations": {
                "l2": 1e-5,
                "augmentation": {
                    "name": "FlipLR",
                    "fn": tf.image.random_flip_left_right,
                    # factor is the estimated amount of augmentation
                    # that "fn" introduces.
                    # In this case, "fn" doubles the training set size
                    # Thus, an epoch is now seen as the original training
                    # training set size * 2
                    "factor": 2,
                }
            },
            "gd": {
                "optimizer": tf.train.AdamOptimizer,
                "args": {
                    "learning_rate": 1e-3,
                    "beta1": 0.9,
                    "beta2": 0.99,
                    "epsilon": 1e-8
                }
            }
        })

During the training of this model, you can monitor the loss trend and the accuracy value using tensorboard.

A new folder is created for you, using some of the representative hyperparameters used:

tensorboard --logdir "log/VGG/CIFAR-10_Adam_l2=1e-05_fliplr/"

as you can see, a folder for the model is created and the hyperparameter used to train it are added next, as subfolder.

This means that if you change the Optimizer (from ADAM to MomentumOptimizer) or you add a comment, or you change the l2 regularization parameter, ecc, DyTB creates a subfolder into the VGG folder.

This allow you to compare the measured metric with tensorboard, using the model directory as logdir, in this way:

tensorboard --logdir log/VGG

For a more comprehensive guide, just have a look at the DyTB README.md or the python-notebook example.

Another viable option for grid search with Tensorflow is Tune. It's a scalable framework/tool for hyperparameter tuning, specifically for deep learning/reinforcement learning.

It also takes care of Tensorboard logging and efficient search algorithms (ie, HyperOpt integration and HyperBand) in about 10 lines of Python.

import ray
from ray import tune

def train_tf_model(config):
    model = Model(lr=config["lr"])
    for x, y in dataset:
        accuracy = model.fit(x, y)
        tune.track.log(accuracy=accuracy)

tune.run(
    train_tf_model, 
    config={"lr": tune.grid_search([0.2, 0.4, 0.6])}
)

(Disclaimer: I contribute actively to this project!)

In the video some code is provided here which allows you to recreate their results.

The code runs the models and the plots are generated by tensorboard with the command tensorboard --logdir <log location>. In this case: tensorboard --logdir /tmp/mnist_tutorial.

The excerpt from the video is as follows:

# Try a few learning rates
for learning_rate in [1E-3, 1E-4, 1E-5]:

    for use_two_fc in [True, False]
        for use_two_conv in [True, False]:

            # Construct a hyperparameter string for each one (example: "lr_1E,fc=2,conv=2)
            hparam_str = make_hparam_string(learning_rate, use_two_fc, use_two_conv)

            writer = tf.summaru/FileWriter("/tmp/mnist_tutorial/" + hparam_str)

            # Actually run with the new settings
            mnist(learning_rate, use_two_fully_connected_layers, _use_two_conv_layers, writer)

Check out the github repo for more a detailed explanation of how to set up tensorboard.

I tried to find a guide to choose best hyperparameters by machine learning.

I chose a TensorFlow classification problem and calculated the accuracy by full-factorial hyperparameter grid search. Then I tried to fit a logistic regression and another DNN classifier to 'learn' what set of hyperparameters will be good for my problem.

Results were bit confusing... but it may just work for your particular problem. You can take a look at: https://medium.com/@tirthajyoti/when-machine-learning-tries-to-predict-the-performance-of-machine-learning-6cc6a11bb9bf