R keras model on tfdatasets, repeated column

Question

I am trying to figure out how to use tfdataset in my model, the reason is mentioned in here: Keras predict repeated columns However I am stuck trying to make the code run. Following is one piece of code that I tried to test how tfdatasets works, but it is not working. This code is just trying to use a keras model to predict (or fit) on a dataset with 3 columns (c1,c2,c3) and the target column y.

library(keras)
library(tfaddons)
library(tfdatasets)

alldata <- data.frame(c1=rnorm(100),c2=rnorm(100),c3=rnorm(100), target=rnorm(100))
alldata_xvar <- c("c1","c2","c3")

write.csv(alldata,file = "alldata.csv", row.names=F)

alldata_spec <- csv_record_spec("alldata.csv")

xfeatures <- alldata_xvar
model <- keras_model_sequential()
model %>% 
  layer_dense(units = length(xfeatures), activation = "relu", input_shape = length(xfeatures)) %>% 
  layer_dense(units = 1) 


dataset <- text_line_dataset("alldata.csv", record_spec = alldata_spec) 
dataset %>% dataset_prepare(x = xfeatures,y="target")

predicted <- model %>%predict(dataset)

The error message is:

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: in user code:

    File "/.local/share/r-miniconda/envs/r-reticulate/lib/python3.7/site-packages/keras/engine/training.py", line 1621, in predict_function  *
        return step_function(self, iterator)
    File "/.local/share/r-miniconda/envs/r-reticulate/lib/python3.7/site-packages/keras/engine/training.py", line 1611, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/.local/share/r-miniconda/envs/r-reticulate/lib/python3.7/site-packages/keras/engine/training.py", line 1604, in run_step  **
        outputs = model.predict_step(data)
    File "/.local/share/r-miniconda/envs/r-reticulate/lib/python3.7/site-packages/keras/engine/training.py", line 1572, in predict_step
        return self(x, training=False)
    File "/.local/share/r-miniconda/envs/r-reticulate/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 67, in error_handl

My final goal is to run predict when the dataset has 3 columns (c1,c2,c3) and 100 extra column (RegionVar1,...RegionVar100) that has the same value for all rows. I don't want to have all these 100 columns in my file alldata.csv (which makes the file too big) so I think I could use the dataset_map function to add extra information to the data read from file alldata.csv. In the following code, region[regionvar,] is a vector of length 100 that contains the values of (RegionVar1,...RegionVar100) which I want to be the same for all records (rows).

dataset <- dataset %>% 
  dataset_map(function(record) {
    record$Region <- region[regionvar,]
    record
  })

Following is my code which is obviously not working (since the first piece of code doesn't work):

library(keras)
library(tfaddons)
library(tfdatasets)
numregion <- 50
numvarregion <-  100
alldata <- data.frame(c1=rnorm(100),c2=rnorm(100),c3=rnorm(100), target=rnorm(100))
alldata_xvar <- c("c1","c2","c3")
region <- matrix( rnorm(50*numvarregion), 50, numvarregion) 
regionvar_name1 <- rep("RegionVar",numvarregion)
regionvar_name2 <- seq(1,numvarregion)
regionvar_name <- cbind(regionvar_name1,regionvar_name2)
regionvar_name <- apply(regionvar_name,1,paste,collapse="")
region <- as.data.frame(region)
names(region) <- regionvar_name



region <- array_reshape(as.matrix(region), c(nrow(region), ncol(region)))


write.csv(alldata,file = "alldata.csv", row.names=F)
alldata_spec <- csv_record_spec("alldata.csv")

xfeatures <- c(alldata_xvar,names(region)) 
model <- keras_model_sequential()
model %>% 
  layer_dense(units = length(xfeatures), activation = "relu", input_shape = length(xfeatures)) %>% 
  layer_dense(units = 1) 


regionvar <- 1
dataset <- text_line_dataset("alldata.csv", record_spec = alldata_spec) 
dataset <- dataset %>% 
  dataset_map(function(record) {
    record$Region <- region[regionvar,]
    record
  })
dataset %>% dataset_prepare(x = xfeatures,y="target")
predicted <- model %>%predict(dataset)

Can you give me suggestions as for what I should do to make the code run or is there another way to solve the problem of repeated column in my original link?

R keras model on tfdatasets, repeated column

0 Answers