What's the exact meaning of `waitMsAvg` in JobStatistics?

user_17040531 0
Viewed 90

Here is a part of JSON of my BigQuery job. You can see that waitMsAvg is much higher than other stuffs. I want to understand the exact meaning of this.

...
          "waitRatioAvg": 0.12868179138919197,
          "waitMsAvg": "205214",
          "waitRatioMax": 0.13476867702572207,
          "waitMsMax": "214921",
          "readRatioAvg": 0,
          "readMsAvg": "0",
          "readRatioMax": 0,
          "readMsMax": "0",
          "computeRatioAvg": 0.005780879641822491,
          "computeMsAvg": "9219",
          "computeRatioMax": 0.021879428621593488,
          "computeMsMax": "34892",
          "writeRatioAvg": 0.000007524737574777079,
          "writeMsAvg": "12",
          "writeRatioMax": 0.00023326686481808947,
          "writeMsMax": "372",
...

In this document, it says "Milliseconds the average shard spent waiting to be scheduled.", but what is actually scheduled?

Does it mean that when the stage was about to start, there was not enough idle slots so the scheduler waited until it was able to start?

If I optimize other queries so I could secure some idle slots, then will waitMsAvg decrease?

google-bigquery google-cloud-platform
1 Answers

It is probably true, but its hard to say. All such situation I have seen were caused by hitting limit of slots.

I suggest to try to use Monitoring which has 3 metrics that can help you to find if this is true:

  • Slots allocated
  • Slots allocated by job type
  • Slots available

I think that if you analyze those metrics you will confirm, if you are hitting slot limit. Sometimes the limit is shared by multiply projects, you have to consider this as well.

user_124103320
Related