iPython: placing Python output into awk commands

user_5747500 0
Viewed 214

I'm getting the below behaviour in iPython, when I try to use Python's variables within shell commands.

Everything works fine with most of the shell commands (e.g. cat, head, etc.). Although, I'm not able to make it work with awk:

$ ipython
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: path = '/home/mux/'

In [2]: file = 'test.txt'

In [3]: !echo {path+file}
/home/mux/test.txt

In [4]: !cat {path+file}
Testing shell commands in iPython

In [5]: !awk '{print $1}' {path+file}
awk: fatal: cannot open file `{path+file}' for reading (No such file or directory)

In [6]: full_path = {path+file}

In [7]: full_path
Out[7]: {'/home/mux/test.txt'}

In [8]: !awk '{print $1}' full_path
awk: fatal: cannot open file `full_path' for reading (No such file or directory)

In [9]: !awk '{print $1}' /home/mux/test.txt
Testing

Can anyone explain the different behaviour between awk and the rest of shell commands?

Is there any workaround to make this work?

jupyter-notebook ipython awk shell
1 Answers

awk is using characters such as {, } and $ which have a special meaning in shell commands inlined in iPython code. Depending on their intended use (evaluating a python variable or used for the shell command), you need to escape them by doubling them.

You want:

$ ipython3
Python 3.8.6 (default, Sep 25 2020, 09:36:53) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: path = "/tmp/"

In [2]: file = "test.txt"

In [3]: !echo {path+file}
/tmp/test.txt

In [4]: !cat {path+file}
hello

In [5]: !awk '{{print $$1}}' {path+file}
hello

It gets even worse when you have a complex command with pipes and an awk call late in the pipeline, because the behaviour of preceding commands changes.

In [6]: !echo {path+file} | awk '{print $1}'
{path+file}

In [7]: !echo {path+file} | awk '{{print $$1}}'
/tmp/test.txt

A last word, when you write full_path = {path+file}, you are creating a python set that contains the concatenation of strings.

user_18193000
Related