Improving inference time of my pytorch model after profiling

user_2007740 0
Viewed 73

I'm trying to reduce the inference time of a model I have to put in production. I did not design it. It takes ~200/300ms to infer.

I did the following to profile it :

with profile(activities=[ProfilerActivity.CPU], record_shapes=True) as prof:
      with record_function("model_inference"):
        trainer.test(model=model, datamodule=dm)

Here are the results

enter image description here

Going from there, what should be my next steps ?

It seems the data loading is taking most of the time ? Does it mean my Dataset subclass is slow ?

I can also see that the to(device) method take ~12% of the total cpu time. I will only be using CPUs, can I remove the to(device) calls everywhere as they are useless?

pytorch-lightning pytorch deep-learning optimization
1 Answers

You can increase the num_workers based on number of CPU cores in your system and prefetch_factor to load the data in advance. These settings might improve your data loading speed.

Also, if you're not trianing on GPU then you can set device="CPU".

user_80521670
Related