I'm creating a neural network and i want to use the library torch for its autograd function. Now i can convert my data to a torch_tensor, but as soon as i then add that tensor to a list of other tensors they seem to lose their torch properties (which are needed to calculate the gradient at the end of the feedforward loop).
The reason i want to put the tensors in a list is because i want the number of hidden layers and neurons per hidden layer to be customizable by the end-user. Previously, before i started using torch, i accomplished that by making a list of all the separate weight matrices, the amount of which is determined by a user-provided variable.
This is the main learning function for my neural network:
train <- function(x, y, hidden = 4, layers = 3, rate = 0.01, iterations = 10000) {
d <- ncol(x) + 1
x <- torch_tensor(x)
Wn <- list()
Wn[[1]] <- torch_randn(d, hidden[1], requires_grad = T)
if(layers > 1){
for(j in 2:layers){
Wn[[j]] <- torch_randn(hidden[j-1] + 1, hidden[j], requires_grad = T)
}
}
Wn[[layers + 1]] <- torch_randn(hidden[length(hidden)] + 1, 1, requires_grad = T)
for (i in 1:iterations) {
ff <- feedforward(x, Wn)
Wn <- backpropagate(y, Wn, ff, learn_rate = rate)
}
return(Wn)
}
My feedforward function looks like this:
feedforward <- function(x, Wn) {
h <- list(x)
for(k in 1:length(Wn)){
Zn <- cbind(1, h[[k]]) %*% Wn[[k]]
h[[k + 1]] <- relu(Zn)
}
return(h)
}
with relu <- function(x) { max(0,x) }
Is there any way of making this work? Or should i try to find a different method to do the feedforward function?