Tensorflow model returning insane loss values

mospira 0
Viewed 61

I'm a bit of a beginner, so please bare with me. I have a regression model I'm trying to build. Right now I'm experimenting with a neural network, but I'm getting such extremely weird and exorbitant loss values that I don't know where to proceed from here.

Ultimately, I don't know what's really going wrong here. Am I choosing the wrong loss function? Should I be normalizing the labels as well? But if I do, doesn't that defeat the point of having a predicted value? My code is below.

train_dataset = dataset.sample(frac=0.8, random_state=0)
test_dataset = dataset.drop(train_dataset.index)

train_dataset.describe().transpose()

This returns the following:

[table][1]

With 'feat3' being the value I want to predict, I separate it from the features like so.

train_features = train_dataset.copy() 
test_features = test_dataset.copy()

train_labels = train_features.pop('feat3') 
test_labels = test_features.pop('feat3')

Since the input features are on wildly different scales, I normalize the first layer.

normalizer = tf.keras.layers.Normalization(axis=-1)
normalizer.adapt(np.array(train_features))

To check that my features are normalized, I run the following.

sample= np.array(train_features[:1])

with np.printoptions(precision=2, suppress=True):
  print('First example:', first)
  print()
  print('Normalized:', normalizer(sample).numpy())

Which prints the following.

First example: [[1.14e+05 2.03e-03 1.68e-04 2.68e-03 5.92e+07]
 [6.15e+03 1.28e-02 3.99e-03 6.11e-03 5.20e+07]
 [2.93e+05 3.58e-02 4.60e-03 2.23e-02 1.33e+08]
 [4.14e+06 4.57e-02 1.27e-02 5.27e-03 6.81e+07]
 [7.11e+04 2.54e-02 1.51e-03 1.03e-02 1.06e+08]
 [5.83e+04 1.96e-02 1.15e-03 3.88e-03 4.44e+07]
 [9.22e+04 4.43e-02 7.27e-03 1.78e-02 2.46e+08]
 [3.85e+04 3.05e-03 3.23e-03 6.31e-03 4.32e+07]
 [2.96e+04 5.34e-03 8.48e-03 1.45e-02 6.41e+07]
 [4.42e+06 1.77e-02 1.96e-03 8.83e-03 5.43e+07]]

Normalized: [[-0.2  -0.   -0.05 -0.95 -0.58]
 [-0.22 -0.   -0.04 -0.61 -0.74]
 [-0.17 -0.   -0.03  0.98  1.01]
 [ 0.51 -0.    0.01 -0.69 -0.39]
 [-0.21 -0.   -0.05 -0.2   0.44]
 [-0.21 -0.   -0.05 -0.83 -0.9 ]
 [-0.2  -0.   -0.02  0.54  3.47]
 [-0.21 -0.   -0.04 -0.59 -0.93]
 [-0.22 -0.   -0.01  0.21 -0.47]
 [ 0.56 -0.   -0.04 -0.34 -0.69]]

Now that my data is seemingly normalized, I can go about defining, compiling and running the model.

def build_and_compile_model(norm):
  model = keras.Sequential([
      norm,
      layers.Dense(5, activation='relu'),
      layers.Dense(1)
  ])

  model.compile(loss='mean_squared_error',
                optimizer=tf.keras.optimizers.Adam(0.001),
                metrics=['mse']
                )
  return model

dnn_model = build_and_compile_model(normalizer)

%%time
history = dnn_model.fit(
    train_features,
    train_labels,
    validation_split=0.2,
    epochs=100)

I ran this overnight just to get a feel for the model and what hyperparameters I should begin tweaking. However after 100 epochs I was getting loss values in the order of trillions. I forgot to save the loss graph itself, but I still have the data up to 88 epochs.

Epoch 1/100
97688/97688 [==============================] - 196s 2ms/step - loss: 44331637407744.0000 - mse: 44331637407744.0000 - val_loss: 39456153272320.0000 - val_mse: 39456153272320.0000
Epoch 2/100
97688/97688 [==============================] - 207s 2ms/step - loss: 39772068249600.0000 - mse: 39772068249600.0000 - val_loss: 34276091363328.0000 - val_mse: 34276091363328.0000
Epoch 3/100
97688/97688 [==============================] - 208s 2ms/step - loss: 34089449029632.0000 - mse: 34089449029632.0000 - val_loss: 28933246418944.0000 - val_mse: 28933246418944.0000
Epoch 4/100
97688/97688 [==============================] - 210s 2ms/step - loss: 29220210212864.0000 - mse: 29220210212864.0000 - val_loss: 25191245676544.0000 - val_mse: 25191245676544.0000
Epoch 5/100
97688/97688 [==============================] - 210s 2ms/step - loss: 26195569672192.0000 - mse: 26195569672192.0000 - val_loss: 23218039554048.0000 - val_mse: 23218039554048.0000
Epoch 6/100
97688/97688 [==============================] - 208s 2ms/step - loss: 24764137603072.0000 - mse: 24764137603072.0000 - val_loss: 22252087148544.0000 - val_mse: 22252087148544.0000
Epoch 7/100
97688/97688 [==============================] - 210s 2ms/step - loss: 24091406893056.0000 - mse: 24091406893056.0000 - val_loss: 21790831149056.0000 - val_mse: 21790831149056.0000
Epoch 8/100
97688/97688 [==============================] - 194s 2ms/step - loss: 23749376081920.0000 - mse: 23749376081920.0000 - val_loss: 21535125405696.0000 - val_mse: 21535125405696.0000
Epoch 9/100
97688/97688 [==============================] - 190s 2ms/step - loss: 23670355394560.0000 - mse: 23670355394560.0000 - val_loss: 21398787457024.0000 - val_mse: 21398787457024.0000
Epoch 10/100
97688/97688 [==============================] - 192s 2ms/step - loss: 23671892606976.0000 - mse: 23671892606976.0000 - val_loss: 21302335242240.0000 - val_mse: 21302335242240.0000
Epoch 11/100
97688/97688 [==============================] - 194s 2ms/step - loss: 23643107098624.0000 - mse: 23643107098624.0000 - val_loss: 21243665317888.0000 - val_mse: 21243665317888.0000
Epoch 12/100
97688/97688 [==============================] - 204s 2ms/step - loss: 23688229421056.0000 - mse: 23688229421056.0000 - val_loss: 21195460182016.0000 - val_mse: 21195460182016.0000
Epoch 13/100
97688/97688 [==============================] - 212s 2ms/step - loss: 23761615060992.0000 - mse: 23761615060992.0000 - val_loss: 21153892532224.0000 - val_mse: 21153892532224.0000
Epoch 14/100
97688/97688 [==============================] - 206s 2ms/step - loss: 23789683343360.0000 - mse: 23789683343360.0000 - val_loss: 21122603024384.0000 - val_mse: 21122603024384.0000
Epoch 15/100
97688/97688 [==============================] - 192s 2ms/step - loss: 23909405556736.0000 - mse: 23909405556736.0000 - val_loss: 21096086634496.0000 - val_mse: 21096086634496.0000
Epoch 16/100
97688/97688 [==============================] - 194s 2ms/step - loss: 23928705646592.0000 - mse: 23928705646592.0000 - val_loss: 21078248259584.0000 - val_mse: 21078248259584.0000
Epoch 17/100
97688/97688 [==============================] - 193s 2ms/step - loss: 24077070761984.0000 - mse: 24077070761984.0000 - val_loss: 21051369062400.0000 - val_mse: 21051369062400.0000
Epoch 18/100
97688/97688 [==============================] - 205s 2ms/step - loss: 24119596810240.0000 - mse: 24119596810240.0000 - val_loss: 21031420952576.0000 - val_mse: 21031420952576.0000
Epoch 19/100
97688/97688 [==============================] - 206s 2ms/step - loss: 24162437431296.0000 - mse: 24162437431296.0000 - val_loss: 21012636762112.0000 - val_mse: 21012636762112.0000
Epoch 20/100
97688/97688 [==============================] - 206s 2ms/step - loss: 24269274742784.0000 - mse: 24269274742784.0000 - val_loss: 20997914755072.0000 - val_mse: 20997914755072.0000
Epoch 21/100
97688/97688 [==============================] - 192s 2ms/step - loss: 24290604875776.0000 - mse: 24290604875776.0000 - val_loss: 20983572332544.0000 - val_mse: 20983572332544.0000
Epoch 22/100
97688/97688 [==============================] - 203s 2ms/step - loss: 24282612629504.0000 - mse: 24282612629504.0000 - val_loss: 20971077500928.0000 - val_mse: 20971077500928.0000
Epoch 23/100
97688/97688 [==============================] - 196s 2ms/step - loss: 24423753056256.0000 - mse: 24423753056256.0000 - val_loss: 20960094715904.0000 - val_mse: 20960094715904.0000
Epoch 24/100
97688/97688 [==============================] - 199s 2ms/step - loss: 24412589916160.0000 - mse: 24412589916160.0000 - val_loss: 20954843447296.0000 - val_mse: 20954843447296.0000
Epoch 25/100
97688/97688 [==============================] - 211s 2ms/step - loss: 24430778515456.0000 - mse: 24430778515456.0000 - val_loss: 20943279751168.0000 - val_mse: 20943279751168.0000
Epoch 26/100
97688/97688 [==============================] - 207s 2ms/step - loss: 24548133044224.0000 - mse: 24548133044224.0000 - val_loss: 20933741903872.0000 - val_mse: 20933741903872.0000
Epoch 27/100
97688/97688 [==============================] - 191s 2ms/step - loss: 24621415923712.0000 - mse: 24621415923712.0000 - val_loss: 20927658065920.0000 - val_mse: 20927658065920.0000
Epoch 28/100
97688/97688 [==============================] - 207s 2ms/step - loss: 24642951577600.0000 - mse: 24642951577600.0000 - val_loss: 20920726978560.0000 - val_mse: 20920726978560.0000
Epoch 29/100
97688/97688 [==============================] - 193s 2ms/step - loss: 24680431878144.0000 - mse: 24680431878144.0000 - val_loss: 20909855342592.0000 - val_mse: 20909855342592.0000
Epoch 30/100
97688/97688 [==============================] - 206s 2ms/step - loss: 24765932765184.0000 - mse: 24765932765184.0000 - val_loss: 20906703323136.0000 - val_mse: 20906703323136.0000
Epoch 31/100
97688/97688 [==============================] - 208s 2ms/step - loss: 24757271527424.0000 - mse: 24757271527424.0000 - val_loss: 20900936155136.0000 - val_mse: 20900936155136.0000
Epoch 32/100
97688/97688 [==============================] - 194s 2ms/step - loss: 24902897762304.0000 - mse: 24902897762304.0000 - val_loss: 20899516383232.0000 - val_mse: 20899516383232.0000
Epoch 33/100
97688/97688 [==============================] - 198s 2ms/step - loss: 24909306658816.0000 - mse: 24909306658816.0000 - val_loss: 20894724390912.0000 - val_mse: 20894724390912.0000
Epoch 34/100
97688/97688 [==============================] - 196s 2ms/step - loss: 25056679821312.0000 - mse: 25056679821312.0000 - val_loss: 20890483949568.0000 - val_mse: 20890483949568.0000
Epoch 35/100
97688/97688 [==============================] - 208s 2ms/step - loss: 25119248351232.0000 - mse: 25119248351232.0000 - val_loss: 20887191420928.0000 - val_mse: 20887191420928.0000
Epoch 36/100
97688/97688 [==============================] - 194s 2ms/step - loss: 25132552683520.0000 - mse: 25132552683520.0000 - val_loss: 20879035596800.0000 - val_mse: 20879035596800.0000
Epoch 37/100
97688/97688 [==============================] - 206s 2ms/step - loss: 25170559369216.0000 - mse: 25170559369216.0000 - val_loss: 20873996140544.0000 - val_mse: 20873996140544.0000
Epoch 38/100
97688/97688 [==============================] - 192s 2ms/step - loss: 25265470177280.0000 - mse: 25265470177280.0000 - val_loss: 20867916496896.0000 - val_mse: 20867916496896.0000
Epoch 39/100
97688/97688 [==============================] - 207s 2ms/step - loss: 25264989929472.0000 - mse: 25264989929472.0000 - val_loss: 20863709609984.0000 - val_mse: 20863709609984.0000
Epoch 40/100
97688/97688 [==============================] - 210s 2ms/step - loss: 25348423024640.0000 - mse: 25348423024640.0000 - val_loss: 20861759258624.0000 - val_mse: 20861759258624.0000
Epoch 41/100
97688/97688 [==============================] - 209s 2ms/step - loss: 25412491018240.0000 - mse: 25412491018240.0000 - val_loss: 20857766281216.0000 - val_mse: 20857766281216.0000
Epoch 42/100
97688/97688 [==============================] - 199s 2ms/step - loss: 25512021852160.0000 - mse: 25512021852160.0000 - val_loss: 20854872211456.0000 - val_mse: 20854872211456.0000
Epoch 43/100
97688/97688 [==============================] - 210s 2ms/step - loss: 25598890082304.0000 - mse: 25598890082304.0000 - val_loss: 20849725800448.0000 - val_mse: 20849725800448.0000
Epoch 44/100
97688/97688 [==============================] - 196s 2ms/step - loss: 25732170383360.0000 - mse: 25732170383360.0000 - val_loss: 20846942879744.0000 - val_mse: 20846942879744.0000
Epoch 45/100
97688/97688 [==============================] - 193s 2ms/step - loss: 25751533387776.0000 - mse: 25751533387776.0000 - val_loss: 20848471703552.0000 - val_mse: 20848471703552.0000
Epoch 46/100
97688/97688 [==============================] - 207s 2ms/step - loss: 25765856935936.0000 - mse: 25765856935936.0000 - val_loss: 20843090411520.0000 - val_mse: 20843090411520.0000
Epoch 47/100
97688/97688 [==============================] - 207s 2ms/step - loss: 25792130056192.0000 - mse: 25792130056192.0000 - val_loss: 20838149521408.0000 - val_mse: 20838149521408.0000
Epoch 48/100
97688/97688 [==============================] - 194s 2ms/step - loss: 25951161286656.0000 - mse: 25951161286656.0000 - val_loss: 20832870989824.0000 - val_mse: 20832870989824.0000
Epoch 49/100
97688/97688 [==============================] - 194s 2ms/step - loss: 25963471568896.0000 - mse: 25963471568896.0000 - val_loss: 20832283787264.0000 - val_mse: 20832283787264.0000
Epoch 50/100
97688/97688 [==============================] - 196s 2ms/step - loss: 25974301261824.0000 - mse: 25974301261824.0000 - val_loss: 20828200632320.0000 - val_mse: 20828200632320.0000
Epoch 51/100
97688/97688 [==============================] - 196s 2ms/step - loss: 26060473237504.0000 - mse: 26060473237504.0000 - val_loss: 20823372988416.0000 - val_mse: 20823372988416.0000
Epoch 52/100
97688/97688 [==============================] - 197s 2ms/step - loss: 26134473342976.0000 - mse: 26134473342976.0000 - val_loss: 20822076948480.0000 - val_mse: 20822076948480.0000
Epoch 53/100
97688/97688 [==============================] - 197s 2ms/step - loss: 26234692042752.0000 - mse: 26234692042752.0000 - val_loss: 20822242623488.0000 - val_mse: 20822242623488.0000
Epoch 54/100
97688/97688 [==============================] - 195s 2ms/step - loss: 26303067586560.0000 - mse: 26303067586560.0000 - val_loss: 20822666248192.0000 - val_mse: 20822666248192.0000
Epoch 55/100
97688/97688 [==============================] - 210s 2ms/step - loss: 26345499262976.0000 - mse: 26345499262976.0000 - val_loss: 20819633766400.0000 - val_mse: 20819633766400.0000
Epoch 56/100
97688/97688 [==============================] - 196s 2ms/step - loss: 26390021799936.0000 - mse: 26390021799936.0000 - val_loss: 20816423026688.0000 - val_mse: 20816423026688.0000
Epoch 57/100
97688/97688 [==============================] - 209s 2ms/step - loss: 26469581455360.0000 - mse: 26469581455360.0000 - val_loss: 20814499938304.0000 - val_mse: 20814499938304.0000
Epoch 58/100
97688/97688 [==============================] - 211s 2ms/step - loss: 26474497179648.0000 - mse: 26474497179648.0000 - val_loss: 20807568850944.0000 - val_mse: 20807568850944.0000
Epoch 59/100
97688/97688 [==============================] - 210s 2ms/step - loss: 26635640242176.0000 - mse: 26635640242176.0000 - val_loss: 20804022566912.0000 - val_mse: 20804022566912.0000
Epoch 60/100
97688/97688 [==============================] - 213s 2ms/step - loss: 26638435745792.0000 - mse: 26638435745792.0000 - val_loss: 20800539197440.0000 - val_mse: 20800539197440.0000
Epoch 61/100
97688/97688 [==============================] - 197s 2ms/step - loss: 26767163129856.0000 - mse: 26767163129856.0000 - val_loss: 20797150199808.0000 - val_mse: 20797150199808.0000
Epoch 62/100
97688/97688 [==============================] - 209s 2ms/step - loss: 26794650501120.0000 - mse: 26794650501120.0000 - val_loss: 20796504276992.0000 - val_mse: 20796504276992.0000
Epoch 63/100
97688/97688 [==============================] - 194s 2ms/step - loss: 26790317785088.0000 - mse: 26790317785088.0000 - val_loss: 20794138689536.0000 - val_mse: 20794138689536.0000
Epoch 64/100
97688/97688 [==============================] - 207s 2ms/step - loss: 26820076371968.0000 - mse: 26820076371968.0000 - val_loss: 20793838796800.0000 - val_mse: 20793838796800.0000
Epoch 65/100
97688/97688 [==============================] - 208s 2ms/step - loss: 26943179194368.0000 - mse: 26943179194368.0000 - val_loss: 20792614060032.0000 - val_mse: 20792614060032.0000
Epoch 66/100
97688/97688 [==============================] - 209s 2ms/step - loss: 27034690519040.0000 - mse: 27034690519040.0000 - val_loss: 20791232036864.0000 - val_mse: 20791232036864.0000
Epoch 67/100
97688/97688 [==============================] - 198s 2ms/step - loss: 27094872489984.0000 - mse: 27094872489984.0000 - val_loss: 20788163903488.0000 - val_mse: 20788163903488.0000
Epoch 68/100
97688/97688 [==============================] - 210s 2ms/step - loss: 27150113570816.0000 - mse: 27150113570816.0000 - val_loss: 20785217404928.0000 - val_mse: 20785217404928.0000
Epoch 69/100
97688/97688 [==============================] - 215s 2ms/step - loss: 27170577580032.0000 - mse: 27170577580032.0000 - val_loss: 20782749057024.0000 - val_mse: 20782749057024.0000
Epoch 70/100
97688/97688 [==============================] - 201s 2ms/step - loss: 27200162103296.0000 - mse: 27200162103296.0000 - val_loss: 20779372642304.0000 - val_mse: 20779372642304.0000
Epoch 71/100
97688/97688 [==============================] - 212s 2ms/step - loss: 27310663139328.0000 - mse: 27310663139328.0000 - val_loss: 20777302753280.0000 - val_mse: 20777302753280.0000
Epoch 72/100
97688/97688 [==============================] - 210s 2ms/step - loss: 27276324372480.0000 - mse: 27276324372480.0000 - val_loss: 20774970720256.0000 - val_mse: 20774970720256.0000
Epoch 73/100
97688/97688 [==============================] - 196s 2ms/step - loss: 27305327984640.0000 - mse: 27305327984640.0000 - val_loss: 20774972817408.0000 - val_mse: 20774972817408.0000
Epoch 74/100
97688/97688 [==============================] - 208s 2ms/step - loss: 27356838232064.0000 - mse: 27356838232064.0000 - val_loss: 20771579625472.0000 - val_mse: 20771579625472.0000
Epoch 75/100
97688/97688 [==============================] - 199s 2ms/step - loss: 27469514014720.0000 - mse: 27469514014720.0000 - val_loss: 20768243056640.0000 - val_mse: 20768243056640.0000
Epoch 76/100
97688/97688 [==============================] - 214s 2ms/step - loss: 27585348108288.0000 - mse: 27585348108288.0000 - val_loss: 20768641515520.0000 - val_mse: 20768641515520.0000
Epoch 77/100
97688/97688 [==============================] - 211s 2ms/step - loss: 27720192884736.0000 - mse: 27720192884736.0000 - val_loss: 20766779244544.0000 - val_mse: 20766779244544.0000
Epoch 78/100
97688/97688 [==============================] - 202s 2ms/step - loss: 27763480199168.0000 - mse: 27763480199168.0000 - val_loss: 20763958575104.0000 - val_mse: 20763958575104.0000
Epoch 79/100
97688/97688 [==============================] - 212s 2ms/step - loss: 27823546826752.0000 - mse: 27823546826752.0000 - val_loss: 20762652049408.0000 - val_mse: 20762652049408.0000
Epoch 80/100
97688/97688 [==============================] - 210s 2ms/step - loss: 27889193975808.0000 - mse: 27889193975808.0000 - val_loss: 20763419607040.0000 - val_mse: 20763419607040.0000
Epoch 81/100
97688/97688 [==============================] - 210s 2ms/step - loss: 27935513772032.0000 - mse: 27935513772032.0000 - val_loss: 20758323527680.0000 - val_mse: 20758323527680.0000
Epoch 82/100
97688/97688 [==============================] - 209s 2ms/step - loss: 27935505383424.0000 - mse: 27935505383424.0000 - val_loss: 20757165899776.0000 - val_mse: 20757165899776.0000
Epoch 83/100
97688/97688 [==============================] - 196s 2ms/step - loss: 27985644093440.0000 - mse: 27985644093440.0000 - val_loss: 20755146342400.0000 - val_mse: 20755146342400.0000
Epoch 84/100
97688/97688 [==============================] - 210s 2ms/step - loss: 28002886877184.0000 - mse: 28002886877184.0000 - val_loss: 20751805579264.0000 - val_mse: 20751805579264.0000
Epoch 85/100
97688/97688 [==============================] - 198s 2ms/step - loss: 28104259010560.0000 - mse: 28104259010560.0000 - val_loss: 20749697941504.0000 - val_mse: 20749697941504.0000
Epoch 86/100
97688/97688 [==============================] - 197s 2ms/step - loss: 28112161079296.0000 - mse: 28112161079296.0000 - val_loss: 20748796166144.0000 - val_mse: 20748796166144.0000
Epoch 87/100
97688/97688 [==============================] - 214s 2ms/step - loss: 28229131829248.0000 - mse: 28229131829248.0000 - val_loss: 20741556797440.0000 - val_mse: 20741556797440.0000
Epoch 88/100
95208/97688 [============================>.] - ETA: 4s - loss: 28422577324032.0000 - mse: 28422577324032.0000
deep-learning machine-learning keras tensorflow python
1 Answers

First, let's think why this might be happening given your data. Your model's initial predictions are likely to be rather small. The weights are initialized to be small, and your norm layer squashes the inputs, so those are also small. We also know that at least one of your training labels is roughly 1e9. If we only consider the loss of your model on this one training sample, your MSE should be roughly 1e9 ** 2 / |training set| = 1e18 / 4e6 = 1.5e11 or 150 billion. This loss value is so massive that the resulting gradients will also be gigantic, and your weights will be wildly fluctuating with each update.

You 100% can and should normalize your labels in this case. You can use the same method you used for your features. When you call adapt Your normalizer is going to calculate a mean and variance for your label column. When you want to get live predictions from your model after training, you can simply do mean + variance * model(x) to retrieve un-normalized predictions.

You should also be sure to evaluate the model on your test set while using the means and variances obtained from your training data to noramlize the test features and labels.

A last couple of other things to consider:

  • The distributions of your training features and labels look more exponential than gaussian to me. You might want to log scale the features and labels before training.
  • For tabular data like this, an ensemble method like Random Forest or XGBoost will probably outperform a DNN and be easy to implement out-of-the-box.
  • With so few features and a tiny model like that, you shouldn't have to train for more than a couple of minutes, even with 4M training samples. Even 1 epoch is probably enough.
Nameer Hirschkind0
Related