Logging to a file and adding timestamps to every single command bash script

user_11801256 0
Viewed 43

I'm writing a bash script that integrates with several modules and call several scripts (python, js, etc). I want to log everything (from bash, from the python/js scripts) to one single log file, and add timestamps to it.

So, say my Bash script is called bash-script.sh, my Python script is called python-script.pyand my js script is js-script.js. The log file is out.log

Now, I'll split this question into two.

Part I:

To log every single command in the bash script and add timestamps to it, bash-script.sh begins with the following:

#!/bin/bash
source ~/.bashrc

exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1> out.log 2>&1
exec > >(while IFS= read -r line; do printf '%s %s %s\n' "$(date --rfc-3339=seconds)" "[Bash-script]" "$line"; done 3>out.log)

The problem is, the script is adding timestamps to everything, including newlines. In addition, since I'm calling external scripts, it is adding some symbols that makes the whole thing super ugly. Here's an example of an output:

022-05-18 16:37:01+00:00 [Bash-script] Bash script started!
2022-05-18 16:37:01+00:00 [Bash-script] Stopping all services
2022-05-18 16:37:02+00:00 [Bash-script] ● srv1.service - Service1 
2022-05-18 16:37:02+00:00 [Bash-script]   Drop-In: /etc/systemd/system/
2022-05-18 16:37:06+00:00 [Bash-script]
2022-05-18 16:37:06,232 [MainThread] INFO [python-script] In python-script.py

I want to get rid of the newlines and symbols (●)

In addition, I have a function WaitForServices that I run after starting a list of services and it basically waits (by sleeping x amount of seconds) until all services are up completely. Until then, it keeps printing waiting for services x.

The log entries in the out.log file these 'waiting' prints are just echo'd without a timestamp. Output example:

*Im not even sure it makes sense with the done 3>out.log, pls correct me if it's wrong.

Part II

Since I want to integrate with other scripts such as python and js, and write to the same log, I'm redirecting the output with >> and I'm not sure it is a good practice. I also want the bash script to exit completely if the python or the js scripts returns an error (exit code that diff than 0). That's how I currently do it:

bash-script.sh:

#!/bin/bash
source ~/.bashrc
    
    exec 3>&1 4>&2
    trap 'exec 2>&4 1>&3' 0 1 2 3
    exec 1> out.log 2>&1
    exec > >(while IFS= read -r line; do printf '%s %s %s\n' "$(date --rfc-3339=seconds)" "[Bash-script]" "$line"; done 3>out.log)

function exitCode {
  if [ "$?" != "0" ];then
      echo Exiting...
      exit;
  fi
}

.....code
.....code

PYTHONPATH=.. /path-to-virtual-env/python python-script.py  >> out.log ; exitCode $?

.....more code

sudo node /path-to-js-file/js-script.js  >> out.log ; exitCode $?

.....more code

Now, I'm pretty sure it is stupid but I couldn't find any other way to achieve it. I am using the logging lib in the python script but when I specify filename=out.log (which is in the same dir) nothing really happens.

logging.basicConfig(filename='out.log',format="%(asctime)s %(levelname)s [%(module)s] %(message)s", level=logging.INFO, force=True)

Thank you very much for any help

logging shell linux python bash
1 Answers

I'm not sure if this could work, but what if you had another bash script, which ran the first bash script and piped the output to something like ets, which would automatically timestamp every line.

/path/to/bash-script.sh 2>&1 | ets -f "[%F %T] [bash-script]"
/path/to/python-script.py 2>&1 | ets -f "[%F %T] [python-script]"
/path/to/js-script.js 2>&1 | ets -f "[%F %T] [js-script]"

The current road you're going down is very confusing and complex. This obviously doesn't get rid of blank lines, etc. You'd need a separate script to parse that out later, I'm thinking.

user_22850920
Related