Tensorflow running time detect

user_15545407 0
Viewed 44

I've been training a TF2 model like

class MyModel():
    def __init__(self):

        self.inp = Input(name='inp', shape=[None, self.inp_tgt.n_feat], dtype='float32')
        self.network = MyAttentionModel()
        self.model = Model(inputs=self.inp, outputs=self.network.outp)
        self.model.summary()

    def train(self):

        train_dataset = self.dataset(max_epochs)
        opt = Adam()
        loss = BinaryCrossentropy()
        self.model.compile(loss=loss,optimizer=opt)
        self.model.fit(
            x=train_dataset,
            epochs=max_epochs,
            steps_per_epoch=self.n_iter
            )

The issue is the training precess runs too slow, I've checked each inner function in my model offline, but things seem to work fine seperately.

I'd like to use datetime.now() to check the time used in each part of my model like I use in pytorch. But in tensorflow, program seems not running seriesly, thus print(end_time - start_time) output nothing

There're some method I've tried,

  • print time costed
  • cProfile
  • tensorflow-profiler

cProfile and tensorflow-profiler outputs mainly python operation's time cost, which difficult to find the bottleneck in my model.

Is there any other methods may help if I want to calculate the time cost of each component of the network?

tensorflow python
1 Answers

You can use Callbacks to check the time per each epoch.

class TimeCallback(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.times = []

    def on_epoch_begin(self, epoch, logs={}):
        self.epoch_time_start = time.time()

    def on_epoch_end(self, epoch, logs={}):
        self.times.append(time.time() - self.epoch_time_start)
        print(self.times[epoch])

And then pass this as a callback to the model.fit function like below

cb=TimeCallback()
model1.fit(x=train_batch,validation_data=valid_batch,epochs=10,verbose=2,callbacks=cb)

The output will be as follows

Epoch 1/10
162.60295152664185
57/57 - 163s - loss: 38.3437 - accuracy: 0.5433 - val_loss: 7.4644 - val_accuracy: 0.5890 - 163s/epoch - 3s/step
Epoch 2/10
161.40023803710938
57/57 - 161s - loss: 3.3678 - accuracy: 0.6856 - val_loss: 3.4822 - val_accuracy: 0.6160 - 161s/epoch - 3s/step
user_142906970
Related