Input for read_csv pandas function

user_1076540 0
Viewed 2150

I need to feed to pandas read_csv the output of one command I execute with Popen. 

p = subprocess.Popen(cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
pandas.read_csv(stdout,index_col=0)

But obviously stdout is a string and it's interpreted as a path. In the API documentation it says that "any object with a read() method (such as a file handle or StringIO)" can be an input to the read_csv function. How can I get such an object out of the Popen commmand? The final objective is to not write to disk.

Also when I write to disk the contents of stdout I can see that the csv has double quotes for each line

alvarobrandon$ head csvfile.csv
"1507109453,<,java,12447,a3e9c495869d,docker,9.0.4.130,9.0.2.131,9.0.2.131,9.0.4.130,56182,9092,9092,56182,tcp"
"1507109453,<,java,1244,a3e9c495869d,docker,9.0.4.130,9.0.2.131,9.0.2.131,9.0.4.130,56182,9092,9092,56182,tcp"
"1507109453,<,java,12447,a3e9c495869d,docker,9.0.4.130,9.0.2.131,9.0.2.131,9.0.4.130,56182,9092,9092,56182,tcp"
pandas subprocess python
1 Answers

What you need is to read from stdout and store that data into a file-like StringIO object. Here is a minimal working example.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import io
import subprocess
import pandas

cmd = ('cat', '/tmp/csvfile')
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
csv = io.StringIO(process.stdout.read().decode())
data = pandas.read_csv(csv, index_col=0)
csv.close()

Hope this helps!

EDIT (output isn't really CSV, so we have to sanitize it a bit before parsing):

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import io
import subprocess
import pandas

cmd = ('cat', '/tmp/csvfile')
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
csv = io.StringIO()
for line in process.stdout:
    csv.write(line.decode().strip('"\n') + '\n')
csv.seek(0)
data = pandas.read_csv(csv, index_col=0)
csv.close()
user_84856380
Related