Check TPU workload/utilization

user_8821185 0
Viewed 5186

I am training a model, and when I open the TPU in the Google Cloud Platform console, it shows me the CPU utilization (on the TPU, I suppose). It is really, really, low (like 0.07%), so maybe it is the VM CPU? I am wondering whether the training is really proper or if the TPUs are just that strong.

Is there any other way to check the TPU usage? Maybe with a ctpu command?

google-cloud-tpu google-compute-engine tensorflow google-cloud-platform
3 Answers

I would recommend using the TPU profiling tools that plug into TensorBoard. A good tutorial for install and use of these tools can be found here.

You'll run the profiler while your TPU is training. It will add an extra tab to your TensorBoard with TPU-specific profiling information. Among the most useful:

  • Average step time
  • Host idle time (how much time the CPU spends idling)
  • TPU idle time
  • Utilization of TPU Matrix units

Based on these metrics, the profiler will suggest ways to start optimizing your model to train well on a TPU. You can also dig into the more sophisticated profiling tools like a trace viewer, or a list of the most expensive graph operations.

For some guidelines on performance tuning (in addition to those ch_mike already linked) you can look at the TPU performance guide.

user_102412510

If you are looking at GCP -> Compute Engine -> TPU, you are looking at the correct spot. If you see the monitoring graphs of your associated Compute Engine instance, you’ll see the CPU graph is different.

Currently, it doesn’t seem to be any other way to look for that information, since none of these options provide it:

gcloud compute tpus describe <tpu-name> --zone=<zone>

ctpu status --details

Nor does the TPU API

As whether your training is proper or not, it would be hard to say, you can refer to Using TPU and make sure you are following the guidelines there. Another useful resource would be Improving training speed.

user_101610980 
(vm)$ capture_tpu_profile --tpu=$TPU_NAME  --monitoring_level=2

Setting monitoring_level=2 displays more detailed information:

TPU type: TPU v2
Number of TPU Cores: 8
TPU idle time (lower is better): 0.091%
Utilization of TPU Matrix Units is (higher is better): 10.7%
Step time: 1.95 kms (avg), 1.90kms (minute), 2.00 kms (max)
Infeed percentage: 87.5% (avg). 87.2% (min), 87.8 (max)

Ref: https://cloud.google.com/tpu/docs/cloud-tpu-tools#monitor_job

user_19570470
Related