What is the default settings (e.g. hyperparameters) for MatLab's feedforwardnet?

user_5205130 0
Viewed 203

My code is very simple for one layer of 20 neurons:

FFNN = feedforwardnet(20);
FFNN_trained = train(FFNN,x,y);

This can result in a very good performance in a few hundreds of epochs. I want to reproduce it in Pytorch, so I need to know the details, e.g. learning rate, activation function, optimizer, batch size, when to stop, etc. Also, the data splitting for training/validation/testing seems to be random in feedforwardnet.

Where can I find these details for feedforwardnet? How to specify the training/validation/testing in feedforwardnet?

Thank you for the help. I realise that the levenberg-marquardt method is not available in Pytorch.

pytorch matlab
1 Answers

According to the documentations for feedforwardnet, the default setting for this function is to train with the Levenberg-Marquardt backpropagation, aka. damped least-squares -- feedforwardnet(20, 'trainlm') option.

As for the data split, the default seems to be a random 0.7-0.15-0.15 train-validation-test split, using the dividerand function.

From the trainlm page:

trainlm is a network training function that updates weight and bias values according to Levenberg-Marquardt optimization. trainlm is often the fastest backpropagation algorithm in the toolbox, and is highly recommended as a first-choice supervised algorithm, although it does require more memory than other algorithms. Training occurs according to trainlm training parameters, shown here with their default values:

  • net.trainParam.epochs — Maximum number of epochs to train. The default value is 1000.
  • net.trainParam.goal — Performance goal. The default value is 0.
  • net.trainParam.max_fail — Maximum validation failures. The default value is 6.
  • net.trainParam.min_grad — Minimum performance gradient. The default value is 1e-7.
  • net.trainParam.mu — Initial mu. The default value is 0.001.
  • net.trainParam.mu_dec — Decrease factor for mu. The default value is 0.1.
  • net.trainParam.mu_inc — Increase factor for mu. The default value is 10.
  • net.trainParam.mu_max — Maximum value for mu. The default value is 1e10.
  • net.trainParam.show — Epochs between displays (NaN for no displays). The default value is 25.
  • net.trainParam.showCommandLine — Generate command-line output. The default value is false.
  • net.trainParam.showWindow — Show training GUI. The default value is true.
  • net.trainParam.time — Maximum time to train in seconds. The default value is inf.

Validation vectors are used to stop training early if the network performance on the validation vectors fails to improve or remains the same for max_fail epochs in a row. Test vectors are used as a further check that the network is generalizing well, but do not have any effect on training.

From Divide Data for Optimal Neural Network Training:

MATLAB provides 4 built-in functions for splitting data:

  1. Divide the data randomly (default) - dividerand
  2. Divide the data into contiguous blocks - divideblock
  3. Divide the data using an interleaved selection - divideint
  4. Divide the data by index - divideind

You can access or change the division function for your network with this property:

net.divideFcn

Each of the division functions takes parameters that customize its behavior. These values are stored and can be changed with the following network property:

net.divideParam

user_56750940
Related