Extract Bash output to a CSV File for Plotting

sam hassan 0
Viewed 64

I am trying to measure the time it takes for a Kubernetes object to be deployed in a Kubernetes cluster by using the time utility. I am trying to do that severally with a time sleep to get values for multiple simulation of deployments.

This is the script.

#!/bin/bash

function time_check {
    i=$1
    time kubectl apply -f deploy.yml --dry-run=client
}

for i in {1..3}
do 
    time_check $i &
    sleep 2
done

This is the Output

deployment.apps/nginx-raw created (dry run)

real    0m0.421s
user    0m0.359s
sys     0m0.138s
deployment.apps/nginx-raw created (dry run)

real    0m0.359s
user    0m0.443s
sys     0m0.158s
deployment.apps/nginx-raw created (dry run)

real    0m0.138s
user    0m0.412s
sys     0m0.122s
deployment.apps/nginx-raw created (dry run)

real    1.483s
user    0m0.412s
sys     0m0.122s
deployment.apps/nginx-raw created (dry run)

real    1.456s
user    0m0.234s
sys     0m0.567s
deployment.apps/nginx-raw created (dry run)

real    2.345
user    0m0.435s
sys     0m0.123s

Goal

I want to pipe the output and take the first row of each iteration's real 0m0.421s , Then take the number part 0m0.421s and strip the 0m if it's in seconds or just leave it if it's in minutes like 1.483. Also strip the s at the end

The final results should be output in a CSV file to be plotted. The expected output in CSV

real
0.421
0.359
0.138
1.483
1.456
2.345

Add-on

I will do this for another deployment and plot the two times data in a line graph to see the time it takes for each deployment

bash
2 Answers

You are using the shell builtin command time. If you switch to linux's time command you can control the output and get the just the data you want.

$ /usr/bin/time -f  '%e' sleep 1.5
1.50

see man time for more details

you can take the output and pipe it into grep -v deployment | tr '\n' ',' that will strip the dry run lines, and convert the renaming newlines into commas

$ printf "1\njunk\n2\njunk\n3\n" 
1
junk
2
junk
3
$ printf "1\njunk\n2\njunk\n3\n" | grep -v junk | tr '\n' ','
1,2,3, $

this is a quick and dirty way to slice the date. I'm sure there are other solutions as well.

toppk0

I just used a randomized sub-second sleep to get the output stream, but the principle should work.

$: for x in 0 1 2 3 4 5 6 7 8 9; do time sleep 0.$RANDOM; done 2>&1 | 
>    awk 'BEGIN{print "real"}/^real/{ print gensub(/^.*m([0-9.]+)s/,"\\1",1)}'
real
0.266
0.716
0.847
0.251
0.358
0.236
0.669
0.266
0.308
0.856

Explained a bit with inline comments -

$: for x in 0 1 2 3 4 5 6 7 8 9; do # this is just a dump loop
     time sleep 0.$RANDOM           # create some output 
   done 2>&1 |                      # dup time output stderr -> stdout
>    awk >my.csv 'BEGIN{print "real"} # the header
          /^real/{ # match for the lines we want (ignore the rest)
             print gensub(/^.*m([0-9.]+)s/,"\\1",1) #  just print matched part
          }'

separate issue

I want to pipe the output and take the first row of each iteration's real 0m0.421s , Then take the number part 0m0.421s and strip the 0m if it's in seconds or just leave it if it's in minutes like 1.483. Also strip the s at the end

So what you are saying is that if it takes 1.5 seconds you want it to output 1.500, but if it takes 90.0 seconds you want it to output 1.500.

How will you tell which it is without either specifying units or standardizing units?

Paul Hodges0
Related