How to find accuracy when prediction values are continuous and actual vaues are discrete?

user_14717759 0
Viewed 100

I am building a time series prediction model using LSTM My dataset contains datetime column as well as a column named device that displays data in 1 and 0 here is how the dataset looks like

datetime                     device
2020-11-03 06:57:34.328887     0
2020-11-03 06:58:36.177615     1
2020-11-03 06:59:34.328887     1
2020-11-03 07:00:36.177615     1

Basically the device column displays 1 and 0 where 1 stands for the device was in use and 0 stands for the device was not in use, the dataset contains more than 25k records.

What I want to do I am using LSTM to predict if the device will be in use or not in future I have divided the dataset into 80% train, 10% validation, 10% test

My prediction output looks something like this [0.7432404, 0.1035113......] while my actual output is in 1 and 0

So I write a piece of code where I converted the continuous values in my prediction array to 1 and 0 and then checked for the accuracy by comparing it with my actual data

This is the code I wrote

def findAccuracy(y_pred):
    co=(max(y_pred)-min(y_pred))/2
    co=min(y_pred)+co
    
    return co

def AccuracyCheck(co, y_pred, y_act):
    testr=[]
    for i in range(len(y_pred)):
      if (y_pred[i]>=co):
        testr.append(1)
      else:
        testr.append(0)
        
    comp=[]
    for i in range(len(y_pred)):
      if(testr[i]==np.float64(y_act[i])):
        comp.append(1)
      else:
        comp.append(0)  
        
    tr_test=comp.count(1)
    pred=tr_test/len(comp)
    print("Test Accuracy is: ", pred*100)    
        
co=findAccuracy(y_pred)
AccuracyCheck(co, y_pred, y_act)

So basically I created a threshold limit and depending on that limit I converted it to 1 and 0 accordingly. The accuracy fluctuates in between 66 to 70 percent, I probably think that the piece of code I have written is the reason why I am not getting better results. How should I improve the accuracy on this categorical data or if there is any other method/algorithm that I can work on so that I can get desired results

The link to the code where I created the model is here: https://towardsdatascience.com/3-steps-to-forecast-time-series-lstm-with-tensorflow-keras-ba88c6f05237

lstm tensorflow time-series machine-learning python
1 Answers

You can use AUC instead of accuracy to assess the performance, whether it is learning across epochs or not. Based on the ROC curve you can also identify the most suitable threshold for your model.

Also bare in mind categorical data refer to data where features or the input is categorical. Here your output/response variable/y is categorically (binary). Therefore, it is a classification problem and the output you have comes from a logistic function. If you look into logistic function you will see it does not return 0-1, rather bounds the output into a range [0, 1].

Here there is a list of all available scikit-learn metrics. You can see which one suits classification (multi-class) and which of only two classes. Also remember, model evaluation does not look into your feature space. So does not matter if you have used LSTM or any other technique to generate your feature. What is needed is only your y_pred and y_act

user_7028460
Related