Unable to import modules when using Weights and Biases' sweeps

user_10788239 0
Viewed 1342

I'm trying to improve my keras neural network hyperparameters by optimizing them with the weights and biases library (wandb).

Here is my configuration:

method: bayes
metric:
  goal: maximize
  name: Search elo
parameters:
  batch_number:
    distribution: int_uniform
    max: 100
    min: 1
  batch_size:
    distribution: int_uniform
    max: 1024
    min: 1
  epochs:
    distribution: int_uniform
    max: 10
    min: 1
  neural_net_blocks:
    distribution: int_uniform
    max: 5
    min: 1
  num_simulations:
    distribution: int_uniform
    max: 800
    min: 1
  pb_c_base:
    distribution: int_uniform
    max: 25000
    min: 15000
  pb_c_init:
    distribution: uniform
    max: 3
    min: 1
  root_dirichlet_alpha:
    distribution: uniform
    max: 4
    min: 0
  root_exploration_fraction:
    distribution: uniform
    max: 1
    min: 0
program: ../Main.py

However, when I run wandb agent arkleseisure/projectname/sweepcode, I get this error, repeated every time a sweep launches.

2020-09-13 12:15:02,188 - wandb.wandb_agent - INFO - Running runs: ['klawqpqv']
2020-09-13 12:15:02,189 - wandb.wandb_agent - INFO - Cleaning up finished run: klawqpqv
2020-09-13 12:15:03,063 - wandb.wandb_agent - INFO - Agent received command: run
2020-09-13 12:15:03,063 - wandb.wandb_agent - INFO - Agent starting run with config:
        batch_number: 75
        batch_size: 380
        epochs: 10
        neural_net_blocks: 4
        num_simulations: 301
        pb_c_base: 17138
        pb_c_init: 1.5509741790555416
        root_dirichlet_alpha: 2.7032316257955133
        root_exploration_fraction: 0.5768106739703028
2020-09-13 12:15:03,245 - wandb.wandb_agent - INFO - About to run command: python ../Main.py --batch_number=75 --batch_size=380 --epochs=10 --neural_net_blocks=4 --num_simulations=301 --p
b_c_base=17138 --pb_c_init=1.5509741790555416 --root_dirichlet_alpha=2.7032316257955133 --root_exploration_fraction=0.5768106739703028
Traceback (most recent call last):
  File "../Main.py", line 3, in <module>
    import numpy
ModuleNotFoundError: No module named 'numpy'

The sweep crashes after three failed attempts, and I was wondering what I am doing wrong. Surely when W & B is made for machine learning projects, it must be possible to import numpy, so what can I change. My code before that point just imports other files from my project. When I run the code normally, it doesn't crash, but executes perfectly ordinarily.

modulenotfounderror keras numpy
1 Answers

The most likely problem you are running into is that wandb agent is running the python script with a different python interpreter than you were intending.

The solution is to specify the python interpreter by adding something like to the sweep configuration (where python3 is the interpreter you wish to use):

command:
  - ${env}
  - python3
  - ${program}
  - ${args}

This is feature is documented at: https://docs.wandb.com/sweeps/configuration#command

And there is a FAQ for setting the python interpreter at: https://docs.wandb.com/sweeps/faq#sweep-with-custom-commands

To understand a bit more about what is going on you can look at the debugging line that you posted that says: "About to run command:"

python ../Main.py --batch_number=75 --batch_size=380 --epochs=10 --neural_net_blocks=4 --num_simulations=301 --pb_c_base=17138 --pb_c_init=1.5509741790555416 --root_dirichlet_alpha=2.7032316257955133 --root_exploration_fraction=0.5768106739703028

By default wandb agent uses a python interpreter named python. This allows users to customize their environment so python points to their interpreter of choice by using pyenv, virtualenv or other tools.

If you typically run commands with the command-line python2 or python3, you can customize how the agent executes your program by specifying the command key in your configuration file as described above. Alternatively, if your program is executable and your python interpreter is in the first line of your script using #!/usr/bin/env python3 syntax, you can set your command array to be:

command:
  - ${env}
  - ${program}
  - ${args}
user_141446780
Related