How to insert long text from .txt file to a DataFrame

user_14272509 0
Viewed 185

I am building a CNN model, and I have to analyze its training history per epoch. Epoch history looks like this. This history is saved on a .txt file.

Epoch 1/300
11/11 [==============================] - 4s 182ms/step - loss: 8.3641 - accuracy: 0.1382 - f1_m: 0.0676 - precision_m: 0.2398 - recall_m: 0.0408 - lr: 1.0000e-04 - val_loss: 8.6183 - val_accuracy: 0.0952 - val_f1_m: 0.0630 - val_precision_m: 0.1872 - val_recall_m: 0.0379 - val_lr: 1.0000e-04
Epoch 2/300
11/11 [==============================] - 1s 129ms/step - loss: 6.1399 - accuracy: 0.5581 - f1_m: 0.5011 - precision_m: 0.7591 - recall_m: 0.3781 - lr: 1.0000e-04 - val_loss: 6.6084 - val_accuracy: 0.4230 - val_f1_m: 0.1302 - val_precision_m: 0.6426 - val_recall_m: 0.0725 - val_lr: 1.0000e-04
.
.
Epoch 300/300
11/11 [==============================] - 1s 132ms/step - loss: 0.7877 - accuracy: 0.9913 - f1_m: 0.9911 - precision_m: 0.9928 - recall_m: 0.9894 - lr: 1.0000e-04 - val_loss: 1.7068 - val_accuracy: 0.8314 - val_f1_m: 0.8347 - val_precision_m: 0.8433 - val_recall_m: 0.8263 - val_lr: 1.0000e-04

To analyze epoch history, I need to populate those datas to a table. In Python, I think it is a good idea to use DataFrame. Also, at the end of DataFrame, I also want to count the average value per column. For illustration, I want the DataFrame to look like this image below: (I make this using Ms. Excel) enter image description here

What I want to ask is:

  1. How to do substring on .txt file to take all datas for DataFrame column? As far as I know, DataFrame needs columns before it can take a data, and I currently do not know how to get it from my .txt file.
  2. If question number 1 is possible to do, how to store the values inside a Python variable, then populate it inside a DataFrame?
pandas text-files dataframe python
1 Answers

I'm assuming that you can read the .txt file and save it to a variable like text. Then you could do the following:

import pandas as pd


text = '''Epoch 1/300
11/11 [==============================] - 4s 182ms/step - loss: 8.3641 - accuracy: 0.1382 - f1_m: 0.0676 - precision_m: 0.2398 - recall_m: 0.0408 - lr: 1.0000e-04 - val_loss: 8.6183 - val_accuracy: 0.0952 - val_f1_m: 0.0630 - val_precision_m: 0.1872 - val_recall_m: 0.0379 - val_lr: 1.0000e-04
Epoch 2/300
11/11 [==============================] - 1s 129ms/step - loss: 6.1399 - accuracy: 0.5581 - f1_m: 0.5011 - precision_m: 0.7591 - recall_m: 0.3781 - lr: 1.0000e-04 - val_loss: 6.6084 - val_accuracy: 0.4230 - val_f1_m: 0.1302 - val_precision_m: 0.6426 - val_recall_m: 0.0725 - val_lr: 1.0000e-04
Epoch 3/300
11/11 [==============================] - 1s 132ms/step - loss: 0.7877 - accuracy: 0.9913 - f1_m: 0.9911 - precision_m: 0.9928 - recall_m: 0.9894 - lr: 1.0000e-04 - val_loss: 1.7068 - val_accuracy: 0.8314 - val_f1_m: 0.8347 - val_precision_m: 0.8433 - val_recall_m: 0.8263 - val_lr: 1.0000e-04
'''

data = []
for i, line in enumerate(text.split('\n')[1::2], start=1):
    obj = {}
    obj['epoch'] = i
    for x in line.split(' - ')[2:]:
        k, v = x.split(':')
        obj[k] = v.lstrip()
    data.append(obj)

df = pd.DataFrame(data).astype(float)

Printing df, we got:

|    |   epoch |   loss |   accuracy |   f1_m |   precision_m |   recall_m |     lr |   val_loss |   val_accuracy |   val_f1_m |   val_precision_m |   val_recall_m |   val_lr |
|---:|--------:|-------:|-----------:|-------:|--------------:|-----------:|-------:|-----------:|---------------:|-----------:|------------------:|---------------:|---------:|
|  0 |       1 | 8.3641 |     0.1382 | 0.0676 |        0.2398 |     0.0408 | 0.0001 |     8.6183 |         0.0952 |     0.063  |            0.1872 |         0.0379 |   0.0001 |
|  1 |       2 | 6.1399 |     0.5581 | 0.5011 |        0.7591 |     0.3781 | 0.0001 |     6.6084 |         0.423  |     0.1302 |            0.6426 |         0.0725 |   0.0001 |
|  2 |       3 | 0.7877 |     0.9913 | 0.9911 |        0.9928 |     0.9894 | 0.0001 |     1.7068 |         0.8314 |     0.8347 |            0.8433 |         0.8263 |   0.0001 |

I like to store data into a list of dicts, because DataFrame will take care of the columns's name and values. See how data is:

[{'epoch': 1,
  'loss': '8.3641',
  'accuracy': '0.1382',
  'f1_m': '0.0676',
  'precision_m': '0.2398',
  'recall_m': '0.0408',
  'lr': '1.0000e-04',
  'val_loss': '8.6183',
  'val_accuracy': '0.0952',
  'val_f1_m': '0.0630',
  'val_precision_m': '0.1872',
  'val_recall_m': '0.0379',
  'val_lr': '1.0000e-04'},
 {'epoch': 2,
  'loss': '6.1399',
  'accuracy': '0.5581',
  'f1_m': '0.5011',
  'precision_m': '0.7591',
  'recall_m': '0.3781',
  'lr': '1.0000e-04',
  'val_loss': '6.6084',
  'val_accuracy': '0.4230',
  'val_f1_m': '0.1302',
  'val_precision_m': '0.6426',
  'val_recall_m': '0.0725',
  'val_lr': '1.0000e-04'},
 {'epoch': 3,
  'loss': '0.7877',
  'accuracy': '0.9913',
  'f1_m': '0.9911',
  'precision_m': '0.9928',
  'recall_m': '0.9894',
  'lr': '1.0000e-04',
  'val_loss': '1.7068',
  'val_accuracy': '0.8314',
  'val_f1_m': '0.8347',
  'val_precision_m': '0.8433',
  'val_recall_m': '0.8263',
  'val_lr': '1.0000e-04'}]
user_111225130
Related