How to avoid data averaging when logging to metric across multiple runs?

aweeeezy 0
Viewed 35

I'm trying to log data points for the same metric across multiple runs (wandb.init is called repeatedly in between each data point) and I'm unsure how to avoid the behavior seen in the attached screenshot...

Instead of getting a line chart with multiple points, I'm getting a single data point with associated statistics. In the attached e.g., the 1st data point was generated at step 1,470 and the 2nd at step 2,940...rather than seeing two points, I'm instead getting a single point that's the average and appears at step 2,205. enter image description here

My hunch is that using the resume run feature may address my problem, but even testing out this hunch is proving to be cumbersome given the constraints of the system I'm working with...

Before I invest more time in my hypothesized solution, could someone confirm that the behavior I'm seeing is, indeed, the result of logging data to the same metric across separate runs without using the resume feature?

If this is the case, can you confirm or deny my conception of how to use resume?

Initial run:

  1. run = wandb.init()
  2. wandb_id = run.id
  3. cache wandb_id for successive runs

Successive run:

  1. retrieve wandb_id from cache
  2. wandb.init(id=wandb_id, resume="must")

Is it also acceptable / preferable to replace 1. and 2. of the initial run with:

  1. wandb_id = wandb.util.generate_id()
  2. wandb.init(id=wandb_id)
wandb
2 Answers

It looks like you’re grouping runs so that could be why it’s appearing as averaging across step - this might not be the case but it’s worth trying. Turn off grouping by clicking the button in the centre above your runs table on the left - it’s highlighted in purple in the image below.

enter image description here

Both of the ways you’re suggesting resuming runs seem fine.

Scott Condron0

My hunch is that using the resume run feature may address my problem,

Indeed, providing a cached id in combination with resume="must" fixed the issue. enter image description here

Corresponding snippet:

import wandb

# wandb run associated with evaluation after first N epochs of training.
wandb_id = wandb.util.generate_id()
wandb.init(id=wandb_id, project="alrichards", name="test-run-3/job-1", group="test-run-3")
wandb.log({"mean_evaluate_loss_epoch": 20}, step=1)
wandb.finish()

# wandb run associated with evaluation after second N epochs of training.
wandb.init(id=wandb_id, resume="must", project="alrichards", name="test-run-3/job-2", group="test-run-3")
wandb.log({"mean_evaluate_loss_epoch": 10}, step=5)
wandb.finish()
aweeeezy0
Related