How can I parse CSV files on the Linux command line?

Joel_60956 0
Viewed 61643

How can I parse CSV files on the Linux command line?

To do things like:

csvparse -c 2,5,6 filename

to extract fields from columns 2, 5 and 6 from all rows.

It should be able to handle the csv file format: https://www.rfc-editor.org/rfc/rfc4180 which means quoting fields and escaping inner quotes as appropriate, so for an example row with 3 fields:

field1,"field, number ""2"", has inner quotes and a comma",field3

so that if I request field 2 for the row above I get:

field, number "2", has inner quotes and a comma

I appreciate that there are numerous solutions, Perl, Awk (etc.) to this problem but I would like a native bash command line tool that does not require me to invoke some other scripting environment or write any additional code(!).

csv unix linux
12 Answers

My FOSS CSV stream editor CSVfix does exactly what you want. There is a binary installer for Windows, and a compilable version (via a makefile) for UNIX/Linux.

Admin1

As suggested by @Jonathan in a comment, there is a module for python that provides the command line tool csvfilter. It works like cut, but properly handles CSV column quoting:

csvfilter -f 1,3,5 in.csv > out.csv

If you have python (and you should), you can install it simply like this:

pip install csvfilter

More info at https://github.com/codeinthehole/csvfilter/

user_2559610

Try crush-tools, they are great at manipulating delimited data. It sounds like exactly what you're looking for.

jmanning2k_14800

My gut reaction would be to write a script wrapper around Python's csv module (if there isn't already such a thing).

Jeremy_Cantrell_188660

I wrote one of these tools too (UNIX only) called csvprintf. It can also converts to XML in an online fashion.

user_2638010

A quick google reveals an awk script that seems to handle csv files.

user_413380

This sounds like a job for awk.

You will most likely need to write your own script for your specific needs, but this site has some dialogue about how to go about doing this.

You could also use the cut utility to strip the fields out.

Something like:

cut -f 2,5,6 -d , filename

where the -f argument is the field you want and -d is the delimeter you want. You could then sort these results, find the unique ones, or use any other bash utility. There is a cool video here about working with CSV files from the command line. Only about a minute, I'd take a look.

However, I guess you could group the cut utility with awk and not want to use it. I don't really know what exactly you mean by native bash command though, so I'll still suggest it.

user_390360
Related