Can SAC be used instead PPO in Cartpole example?

user_14233195 0
Viewed 494

I'm studying AzureML RL with example codes.

I could run cartpole example (cartpole_ci.ipynb) which trains the PPO model on compute instance.

I tried SAC instead of PPO by changing training_algorithm = "PPO" to training_algorithm = "SAC" but it failed with the message below.

ray.rllib.utils.error.UnsupportedSpaceException: Action space Discrete(2) is not supported for SAC.

Has someone tried SAC algorithm on AzureML RL and did it work?

azure-machine-learning-service
1 Answers

AzureML RL does support SAC Discrete Actions but not parametric and I have confirmed it in the doc - https://docs.ray.io/en/latest/rllib-algorithms.html#feature-compatibility-matrix

Are you following the code sample?

from azureml.contrib.train.rl import ReinforcementLearningEstimator, Ray

training_algorithm = "PPO" rl_environment = "CartPole-v0"

script_params = {

    # Training algorithm
    "--run": training_algorithm,
    
    # Training environment
    "--env": rl_environment,
    
    # Algorithm-specific parameters
    "--config": '\'{"num_gpus": 0, "num_workers": 1}\'',
    
    # Stop conditions
    "--stop": '\'{"episode_reward_mean": 200, "time_total_s": 300}\'',
    
    # Frequency of taking checkpoints
    "--checkpoint-freq": 2,
    
    # If a checkpoint should be taken at the end - optional argument with no value
    "--checkpoint-at-end": "",
    
    # Log directory
    "--local-dir": './logs' }

training_estimator = ReinforcementLearningEstimator(

    # Location of source files
    source_directory='files',
    
    # Python script file
    entry_script='cartpole_training.py',
    
    # A dictionary of arguments to pass to the training script specified in ``entry_script``
    script_params=script_params,
    
    # The Azure Machine Learning compute target set up for Ray head nodes
    compute_target=compute_target,
    
    # Reinforcement learning framework. Currently must be Ray.
    rl_framework=Ray() )
user_95988010
Related