Why is custom neural network much worse than inbuilt in R with Keras?

Myoujin 0
Viewed 32

I am currently stuck with analyzing the performance of two (almost) identical neural networks built in R with Keras. The first is a custom-built one using the syntax of the official documentation for Tensorflow + Keras with R; the second one is built according to the sequential model type of Tensorflow + Keras with R. A fully runnable example is presented below. The custom neural network especially does not yield good results. The MSE loss is over 80 for a long time, even when epochs are in the hundreds. The sequential built neural network, however, can model the dependence already quite well after a few epochs. Can anybody explain to me the reason for that observation? Thanks in advance.

CUSTOM MODEL
library(keras)
library(tensorflow)

epochs <- 20
batch_size <- 2^4

dataset <- dataset_boston_housing()

c(c(train_data, train_targets), c(test_data, test_targets)) %<-% dataset

X_mean <- apply(train_data, 2, mean)
X_std <- apply(train_data, 2, sd)

train_data <- scale(train_data, center = X_mean, scale = X_std)
test_data <- scale(test_data, center = X_mean, scale = X_std)

set_random_seed(1234)

model_input <- layer_input(shape = dim(train_data)[[2]]) 
model_output <- model_input %>%
  layer_dense(units = 64, activation = "relu", kernel_initializer = initializer_he_uniform()) %>%
  layer_dense(units = 64, activation = "relu", kernel_initializer = initializer_he_uniform()) %>%
  layer_dense(units = 1)

my_model <- keras_model(model_input,model_output)

m_optimizer <- optimizer_adam(learning_rate=0.001)

my_mse <- function(y_true, y_pred){
  loss <- mean((y_true-y_pred)^2)
  return(loss)
}

train <- function(X,y,epochs) {
  
  steps <- nrow(X)%/%batch_size
  
  for (epoch in 1:epochs) {
    shuffle_ind <- sample.int(nrow(X)) 
    
    X <- X[shuffle_ind,]
    y <- y[shuffle_ind]
    
    total_loss <- 0
    
    start_index <- 1
    end_index <- batch_size  
    
    for (step in 1:steps) {
      
      real_values <- X[start_index:end_index,]
      with(tf$GradientTape() %as% g_tape,{ 
             y_pred <- my_model(real_values,training=TRUE)
             y_true <- y[start_index:end_index]
             
             m_loss<- my_mse(y_true=y_true, y_pred=y_pred)
             m_gradient <- g_tape$gradient(m_loss, my_model$trainable_variables)
             m_optimizer$apply_gradients(purrr::transpose(list(m_gradient, my_model$trainable_variables)))
           })

      total_loss <- total_loss + m_loss

      if(step==steps){
        average_loss <- total_loss/steps
        cat("loss: ", average_loss$numpy(), "\n")
      }
      
      start_index <- start_index + batch_size
      end_index <- end_index + batch_size
      
    }
  }
}

train(train_data,train_targets,epochs)
SEQUENTIAL MODEL
model <- keras_model_sequential() %>%
  layer_dense(units = 64, activation = "relu", kernel_initializer = initializer_he_uniform(),input_shape = dim(train_data)[[2]]) %>%
  layer_dense(units = 64, activation = "relu", kernel_initializer = initializer_he_uniform()) %>%
  layer_dense(units = 1)
  
model <- model %>% compile(
  loss = my_mse, 
  optimizer = optimizer_adam(lr=0.001))    

history <- model %>% fit(
  train_data, train_targets,
  epochs = epochs, batch_size = batch_size,
)

Edit: Please note, that even substituting the sequential model type "keras_model_sequential()" into the custom-built model type in "my_model" will not change the issue.

keras tensorflow r
1 Answers

I actually have found the solution myself; even though I do not know exactly why that works. I have recreated the problem in Python and ran into the same issues again. As Python is a little bit more unforgiving when it comes to data types, I have found that there are some issues with data conversions, as for some unknown reason to me y_pred and y_true are float tensors and tensorflow expects double tensors. (Error message: "input #1(zero-based) was expected to be a double tensor but is a float tensor [Op:Sub]") Hence, I changed the tensor type, flattened it using Keras, and now it works. So all you have to do is replace the custom loss function with the following:

my_mse <- function(y_true, y_pred){
  y_true <- k_flatten(as_tensor(y_true, dtype = tf$float16))
  y_pred <- k_flatten(as_tensor(y_pred, dtype = tf$float16))
  loss <- mean((y_true-y_pred)^2)
  return(loss)
}

Maybe that helps somebody in the future.

Myoujin0
Related