I am provisioning my Infracture with Terraform and I am using xyz.sh bash script which consists my Deeplearning Model training over GPU Machine.
My question is, How will I get the logs/finishing time of xyz.sh bash script without ssh into the machine? if not possible then, if I will ssh into the machine so, How can i check that the script is still running or finished
When you use user_data for an EC2 instance, what happens internally is that Terraform sends that string to the EC2 API and then the EC2 infrastructure makes that string available to the instance via the Instance metadata and user data API.
How (and whether) that string is used by the EC2 instance is then dependent on what software you have installed in the EC2 instance. A typical configuration for common Linux distribution AMIs is to have cloud-init installed and configured to run on first boot. If you are using an AMI with cloud-init then it will be cloud-init that retrieves the user_data string from the EC2 endpoint and executes it as a script (or, other interpretations), and so cloud-init is the system responsible for emitting any logs that result from that process.
You can read more about debugging cloud-init in Testing and debugging cloud-init, which mentions that cloud-init writes logs to /var/log/cloud-init.log by default (some Linux distributions may customize this) and that you can use the cloud-init analyze subcommand to retrieve information from that log file.
Terraform's involvement in this process is only to send the given user_data string to the EC2 API, so Terraform has no visibility into what happens after the instance is created. Unless the script you submit includes a step to report its progress somewhere, there is no built-in way to determine that other than to inspect the cloud-init log file from within the EC2 instance itself.
You can run command
ls -la /var/log/cloud*
it will out put few of log related to user data : (in my case, I use Ali Cloud, so it shows as) :
Then you need to identify which one is your userdata, in my case the /var/log/cloud-init-output.log is my all userdata output will be stored.
Other cloud providers might be a little different but the concept should be the same because most of the cloud are using same cloud-init library https://cloud-init.io/
Note: you need to ssh into server.
