How to parse a CSV in a Bash script?

user_62032 0
Viewed 135482

I am trying to parse a CSV containing potentially 100k+ lines. Here is the criteria I have:

  1. The index of the identifier
  2. The identifier value

I would like to retrieve all lines in the CSV that have the given value in the given index (delimited by commas).

Any ideas, taking in special consideration for performance?

csv shell bash
12 Answers

Parsing CSV with primitive text-processing tools will fail on many types of CSV input.

xsv is a lovely and fast tool for doing this properly. To search for all records that contain the string "foo" in the third column:

cat file.csv | xsv search -s 3 foo
user_11745210

Awk (gawk) actually provides extensions, one of which being csv processing.

Assuming that extension is installed, you can use awk to show all lines where a specific csv field matches 123.

Assuming test.csv contains the following:

Name,Phone
"Woo, John",425-555-1212
"James T. Kirk",123

The following will print all lines where the Phone (aka the second field) is equal to 123:

gawk -l csv 'csvsplit($0,a) && a[2] == 123 {print $0}'

The output is:

"James T. Kirk",123

How does it work?

  • -l csv asks gawk to load the csv extension by looking for it in $AWKLIBPATH;
  • csvsplit($0, a) splits the current line, and stores each field into a new array named a
  • && a[2] == 123 checks that the second field is 123
  • if both conditions are true, it { print $0 }, aka prints the full line as requested.
user_14434180
Related