Is it possible to fail an aws_instance creation if script passed into user_data fails to run? e.g., with exit 1?
I have a null_resource that uses depends_on [aws_instance.myVM] to post a JSON to the instance, and I need that depends_on to fail when the user_data script fails.
Thanks!
user_data is handled by software running inside the instance itself, such as cloud-init, and so its processing is asynchronous from the ec2:RunInstances call that Terraform's AWS provider makes to start the instance running. There is therefore no way to feed back status information from the user_data handling because it could potentially be running some time after the EC2 API starts reporting that the instance is "running", depending on where in the boot process it's dealt with.
Also, depends_on is for ordering rather than for handling errors, so a depends_on clause will never change anything about Terraform's error handling.
If you want to run software on your instance synchronously as part of Terraform's create operation then unfortunately the only practical option is to use the remote-exec provisioners. This comes at the expense of some considerable extra complexity, because Terraform must now be able to open an SSH session with the instance and authenticate with it to create a two-way communications channel.
In return for that complexity, Terraform can be the one to run the code in question and so Terraform can detect whether it succeeded or not (using its exit status). If the remote command fails, Terraform will halt further processing, return an error, and mark the instance as tainted so that the next plan will attempt to create it again.
With that said, it may be better to find a different way to achieve your goal that doesn't require coupling the Terraform run directly to the state of the software running in the virtual machine. It's not usually expected that a Terraform configuration should need to both start up a virtual machine and interact with software running in that virtual machine in the same operation. This could be a good point for some system decomposition, where you'd have one Terraform configuration that declares that the virtual machine should exist and then a second configuration that assumes that the virtual machine exists and takes actions against it. You can then run whatever other software you need to run in between those two, in order to check whether the instance started up successfully.
One not so obvious way I am doing this is I register the instance as a consul service at the end of user data. If the service never arrives, downstream polling for arrival of the consul service can also timeout and fail. This could be located in another local exec function which could be used as a dependency, or it could be in other instance user data as well.
Perhaps a better way would be to use the consul KV store though, and not to just rely on service arrival. The KV store could be used to provide more information than just a binary state.
