TF LSTM multiple step predict seems to be wrong

If I understand correctly, you are trying to operate iterative prediction. so given in input a whole sequence we use it to predict the next values. we add the prediction at the end and repeat the process.

The piece of code that computes this operation is a bit confused. I try to rewrite it

plist = x_test[-2]
for i in range(20):
    temp = model.predict(plist.reshape(-1,20,1))
    plist = np.append(plist[1:],temp[0])

we start from x_test[-2] because our aims is to predict x_test[-1]. in the end, the result is shown below

plt.plot(x_test[-1], label='true')
plt.plot(plist, label='pred')
plt.legend()

the procedure is now correct (also without reverse scaling)

I went ahead and read over your data a bit. Some comments on your training and model:

1) You specifically trained an LSTM to predict one value into the future from 20 past values. Why would you expect that model to be able to extrapolate to predicting another 19 values from the first one value? Your model was not trained to do that and doesn't know that is important.

2) Your model is extremely small. Your model has 491 variables (model.summary()), which is very tiny. How many features do you think influences the stock market? How many variables do you think model each feature? Most deep learning is going to take millions of variables to learn something complicated like the stock market. In addition this is a highly complicated field that is taking millions of people around the world to research.

3) You have barely done any kind of training for how complicated your data is. You ran 100 epochs, or about 250 seconds of training on a GPU.

4) Your model is EXTREMELY overfit. I think you are making the mistake in thinking that "low loss" == "good predicting model", this is simply not true. Here is an example I google'd for that shows what I mean,

https://towardsdatascience.com/regularization-the-path-to-bias-variance-trade-off-b7a7088b4577

https://miro.medium.com/max/552/1%2ARN6fYYl_Z_n9R3cr1zpcTA@2x.png

In this example, the trained model has essentially zero loss and predicts perfectly on the input data. But the model clearly knows nothing about the underlying feature curve that drives the input data with some noise. Hence, if you tried to predict the next value with the model, it will be extremely inaccurate.

5) I think you are misunderstanding what training a model does. When you train a model, you are converging to a "common" answer that produces a low loss. A model is never going to find some extreme example of weights that predicts really well; it is only going to find and slowly converge to a common solution that yields low loss. If for example, slightly adjusting your weights towards zero, makes the loss smaller, then that will be repeatedly emphasized during your training. If predicting a uniform linear answer produces the least loss amongst a seemingly random set of data, then that will be reinforced over and over again during training. In general (like I commented above), when a model is faced with uniformly random information, it has a tendency to favor the average of the information (because that produces less loss from batch to batch to batch).

I believe the code you have provided and the results you have gotten are expected.

EDIT:

Here are some examples of time-series data models (are they good, who knows, that depends on your data):

# Basic LSTM that attempts to produce a prediction vector from a sequence of time data. 
model = tf.keras.Sequential([
    tf.keras.layers.LSTM(20), 
])

# Maybe throw a few LSTM layers to help learn the information, add some dimensions. 
model = tf.keras.Sequential([
    tf.keras.layers.LSTM(5,return_sequences=True), 
    tf.keras.layers.LSTM(50,return_sequences=True),
    tf.keras.layers.LSTM(20),
])

# Maybe some Dense layers should be used to re-arrange information. 
model = tf.keras.Sequential([
    tf.keras.layers.LSTM(20), 
    tf.keras.layers.Dense(500),
    tf.keras.layers.Dense(20),
])

Maybe throw in some activations, relu might be useful (kind of like giving the layer a garbage can to remove useless values). Or maybe some sigmoid (keep values small) or softmax (treat values like probabilities). You could also throw in some Dropout layers to prevent overfitting (keep your model on it's toes, don't let it get settled to quickly). Also, normalizing data can be useful. If you find the weights grow to quickly, add some regularization to force the model to use small weights while trying to learn the data.

model = tf.keras.Sequential([
    tf.keras.layers.LSTM(5,return_sequences=True,activation='relu'), 
    tf.keras.layers.Dropout(0.10),
    tf.keras.layers.LSTM(50,return_sequences=True,activation='relu'),
    tf.keras.layers.Dropout(0.10),
    tf.keras.layers.LSTM(20),
])

If you think the learning is going bad, switch from "adam" to another learning algorithm. Maybe your "Mean Squared Error" loss function is too harsh to large errors, maybe slow down the learning; try "logcosh" instead. Run it for 24 hours straight, preferably use a GPU.

If the LSTM layers are too slow. Maybe switch to using some Attention or Residual layers instead (got to google for this custom ideas).

You could always try a GRU layer instead of LSTM. Has about the same performance, but trains a little faster.