Dealing with commas in a CSV file

user_55102 0
Viewed 602479

I am looking for suggestions on how to handle a csv file that is being created, then uploaded by our customers, and that may have a comma in a value, like a company name.

Some of the ideas we are looking at are: quoted Identifiers (value "," values ","etc) or using a | instead of a comma. The biggest problem is that we have to make it easy, or the customer won't do it.

csv
28 Answers

There's actually a spec for CSV format, RFC 4180 and how to handle commas:

Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes.

http://tools.ietf.org/html/rfc4180

So, to have values foo and bar,baz, you do this:

foo,"bar,baz"

Another important requirement to consider (also from the spec):

If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote. For example:

"aaa","b""bb","ccc"
Corey_Trager_93280

As others have said, you need to escape values that include quotes. Here’s a little CSV reader in C♯ that supports quoted values, including embedded quotes and carriage returns.

By the way, this is unit-tested code. I’m posting it now because this question seems to come up a lot and others may not want an entire library when simple CSV support will do.

You can use it as follows:

using System;
public class test
{
    public static void Main()
    {
        using ( CsvReader reader = new CsvReader( "data.csv" ) )
        {
            foreach( string[] values in reader.RowEnumerator )
            {
                Console.WriteLine( "Row {0} has {1} values.", reader.RowIndex, values.Length );
            }
        }
        Console.ReadLine();
    }
}

Here are the classes. Note that you can use the Csv.Escape function to write valid CSV as well.

using System.IO;
using System.Text.RegularExpressions;

public sealed class CsvReader : System.IDisposable
{
    public CsvReader( string fileName ) : this( new FileStream( fileName, FileMode.Open, FileAccess.Read ) )
    {
    }

    public CsvReader( Stream stream )
    {
        __reader = new StreamReader( stream );
    }

    public System.Collections.IEnumerable RowEnumerator
    {
        get {
            if ( null == __reader )
                throw new System.ApplicationException( "I can't start reading without CSV input." );

            __rowno = 0;
            string sLine;
            string sNextLine;

            while ( null != ( sLine = __reader.ReadLine() ) )
            {
                while ( rexRunOnLine.IsMatch( sLine ) && null != ( sNextLine = __reader.ReadLine() ) )
                    sLine += "\n" + sNextLine;

                __rowno++;
                string[] values = rexCsvSplitter.Split( sLine );

                for ( int i = 0; i < values.Length; i++ )
                    values[i] = Csv.Unescape( values[i] );

                yield return values;
            }

            __reader.Close();
        }
    }

    public long RowIndex { get { return __rowno; } }

    public void Dispose()
    {
        if ( null != __reader ) __reader.Dispose();
    }

    //============================================


    private long __rowno = 0;
    private TextReader __reader;
    private static Regex rexCsvSplitter = new Regex( @",(?=(?:[^""]*""[^""]*"")*(?![^""]*""))" );
    private static Regex rexRunOnLine = new Regex( @"^[^""]*(?:""[^""]*""[^""]*)*""[^""]*$" );
}

public static class Csv
{
    public static string Escape( string s )
    {
        if ( s.Contains( QUOTE ) )
            s = s.Replace( QUOTE, ESCAPED_QUOTE );

        if ( s.IndexOfAny( CHARACTERS_THAT_MUST_BE_QUOTED ) > -1 )
            s = QUOTE + s + QUOTE;

        return s;
    }

    public static string Unescape( string s )
    {
        if ( s.StartsWith( QUOTE ) && s.EndsWith( QUOTE ) )
        {
            s = s.Substring( 1, s.Length - 2 );

            if ( s.Contains( ESCAPED_QUOTE ) )
                s = s.Replace( ESCAPED_QUOTE, QUOTE );
        }

        return s;
    }


    private const string QUOTE = "\"";
    private const string ESCAPED_QUOTE = "\"\"";
    private static char[] CHARACTERS_THAT_MUST_BE_QUOTED = { ',', '"', '\n' };
}
user_45250

The CSV format uses commas to separate values, values which contain carriage returns, linefeeds, commas, or double quotes are surrounded by double-quotes. Values that contain double quotes are quoted and each literal quote is escaped by an immediately preceding quote: For example, the 3 values:

test
list, of, items
"go" he said

would be encoded as:

test
"list, of, items"
"""go"" he said"

Any field can be quoted but only fields that contain commas, CR/NL, or quotes must be quoted.

There is no real standard for the CSV format, but almost all applications follow the conventions documented here. The RFC that was mentioned elsewhere is not a standard for CSV, it is an RFC for using CSV within MIME and contains some unconventional and unnecessary limitations that make it useless outside of MIME.

A gotcha that many CSV modules I have seen don't accommodate is the fact that multiple lines can be encoded in a single field which means you can't assume that each line is a separate record, you either need to not allow newlines in your data or be prepared to handle this.

Robert_Gamble_252220

Put double quotes around strings. That is generally what Excel does.

Ala Eli,

you escape a double quote as two double quotes. E.g. "test1","foo""bar","test2"

user_204710

You can put double quotes around the fields. I don't like this approach, as it adds another special character (the double quote). Just define an escape character (usually backslash) and use it wherever you need to escape something:

data,more data,more data\, even,yet more

You don't have to try to match quotes, and you have fewer exceptions to parse. This simplifies your code, too.

Adam_Jaskiewicz_353220

Add a reference to the Microsoft.VisualBasic (yes, it says VisualBasic but it works in C# just as well - remember that at the end it is all just IL).

Use the Microsoft.VisualBasic.FileIO.TextFieldParser class to parse CSV file Here is the sample code:

 Dim parser As TextFieldParser = New TextFieldParser("C:\mar0112.csv")
 parser.TextFieldType = FieldType.Delimited
 parser.SetDelimiters(",")      

   While Not parser.EndOfData         
      'Processing row             
      Dim fields() As String = parser.ReadFields         
      For Each field As String In fields             
         'TODO: Process field                   

      Next      
      parser.Close()
   End While
user_13274550

Here's a neat little workaround:

You can use a Greek Lower Numeral Sign instead (U+0375)

It looks like this ͵

Using this method saves you a lot of resources too...

user_81088980

I know it's almost 13 years later, but we came across a similar situation where the client inputs us a CSV and has values with commas, there are 2 use cases:

  1. If the client uses a windows Excel client to write the CSV (usually that's the case in windows environment) then commas are automatically added to the value. enter image description here

The actual text value of the CSV:

3786962,1st Meridian Care Services,John,"Person A,Person B, Person C, Person D",Voyager
  1. If the client is sending you the excel programmatically, then he should adhere to RFC4180 and enclose the value with "quotes". example:

Col1, Col2, "a, b, c", Col4

user_114857940

Just use SoftCircuits.CsvParser on NuGet. It will handle all those details for you and efficiently handles very large files. And, if needed, it can even import/export objects by mapping columns to object properties. In addition, my testing showed it averages nearly 4 times faster than the popular CsvHelper.

user_5226630

As this is about general practices let's start from rules of the thumb:

  1. Don't use CSV, use XML with a library to read & write the xml file instead.

  2. If you must use CSV. Do it properly and use a free library to parse and store the CSV files.

To justify 1), most CSV parsers aren't encoding aware so if you aren't dealing with US-ASCII you are asking for troubles. For example excel 2002 is storing the CSV in local encoding without any note about the encoding. The CSV standard isn't widely adopted :(. On the other hand xml standard is well adopted and it handles encodings pretty well.

To justify 2), There is tons of csv parsers around for almost all language so there is no need to reinvent the wheel even if the solutions looks pretty simple.

To name few:

  • for python use build in csv module

  • for perl check CPAN and Text::CSV

  • for php use build in fgetcsv/fputcsv functions

  • for java check SuperCVS library

Really there is no need to implement this by hand if you aren't going to parse it on embedded device.

user_808690

You can read the csv file like this.

this makes use of splits and takes care of spaces.

ArrayList List = new ArrayList();
static ServerSocket Server;
static Socket socket;
static ArrayList<Object> list = new ArrayList<Object>();


public static void ReadFromXcel() throws FileNotFoundException
{   
    File f = new File("Book.csv");
    Scanner in = new Scanner(f);
    int count  =0;
    String[] date;
    String[] name;
    String[] Temp = new String[10];
    String[] Temp2 = new String[10];
    String[] numbers;
    ArrayList<String[]> List = new ArrayList<String[]>();
    HashMap m = new HashMap();

         in.nextLine();
         date = in.nextLine().split(",");
         name = in.nextLine().split(",");
         numbers = in.nextLine().split(",");
         while(in.hasNext())
         {
             String[] one = in.nextLine().split(",");
             List.add(one);
         }
         int xount = 0;
         //Making sure the lines don't start with a blank
         for(int y = 0; y<= date.length-1; y++)
         {
             if(!date[y].equals(""))
             {   
                 Temp[xount] = date[y];
                 Temp2[xount] = name[y];
                 xount++;
             }
         }

         date = Temp;
         name =Temp2;
         int counter = 0;
         while(counter < List.size())
         {
             String[] list = List.get(counter);
             String sNo = list[0];
             String Surname = list[1];
             String Name = list[2];
             for(int x = 3; x < list.length; x++)
             {           
                 m.put(numbers[x], list[x]);
             }
            Object newOne = new newOne(sNo, Name, Surname, m, false);
             StudentList.add(s);
             System.out.println(s.sNo);
             counter++;
         }
user_30146400

The simplest solution I've found is the one LibreOffice uses:

  1. Replace all literal " by ”
  2. Put double quotes around your string

You can also use the one that Excel uses:

  1. Replace all literal " by ""
  2. Put double quotes around your string

Notice other people recommended to do only step 2 above, but that doesn't work with lines where a " is followed by a ,, like in a CSV where you want to have a single column with the string hello",world, as the CSV would read:

"hello",world"

Which is interpreted as a row with two columns: hello and world"

user_3039140 
    public static IEnumerable<string> LineSplitter(this string line, char 
         separator, char skip = '"')
    {
        var fieldStart = 0;
        for (var i = 0; i < line.Length; i++)
        {
            if (line[i] == separator)
            {
                yield return line.Substring(fieldStart, i - fieldStart);
                fieldStart = i + 1;
            }
            else if (i == line.Length - 1)
            {
                yield return line.Substring(fieldStart, i - fieldStart + 1);
                fieldStart = i + 1;
            }

            if (line[i] == '"')
                for (i++; i < line.Length && line[i] != skip; i++) { }
        }

        if (line[line.Length - 1] == separator)
        {
            yield return string.Empty;
        }
    }
user_42033840

I used Csvreader library but by using that I got data by exploding from comma(,) in column value.

So If you want to insert CSV file data which contains comma(,) in most of the columns values, you can use below function. Author link => https://gist.github.com/jaywilliams/385876 

function csv_to_array($filename='', $delimiter=',')
{
    if(!file_exists($filename) || !is_readable($filename))
        return FALSE;

    $header = NULL;
    $data = array();
    if (($handle = fopen($filename, 'r')) !== FALSE)
    {
        while (($row = fgetcsv($handle, 1000, $delimiter)) !== FALSE)
        {
            if(!$header)
                $header = $row;
            else
                $data[] = array_combine($header, $row);
        }
        fclose($handle);
    }
    return $data;
}
user_104515430

I used papaParse library to have the CSV file parsed and have the key-value pairs(key/header/first row of CSV file-value).

here is example that I use:

https://codesandbox.io/embed/llqmrp96pm

it has dummy.csv file in there to have the CSV parsing demo.

I've used it within reactJS though it is easy and simple to replicate in app written with any language.

user_98788840

An example might help to show how commas can be displayed in a .csv file. Create a simple text file as follows:

Save this text file as a text file with suffix ".csv" and open it with Excel 2000 from Windows 10.

aa,bb,cc,d;d "In the spreadsheet presentation, the below line should look like the above line except the below shows a displayed comma instead of a semicolon between the d's." aa,bb,cc,"d,d", This works even in Excel

aa,bb,cc,"d,d", This works even in Excel 2000 aa,bb,cc,"d ,d", This works even in Excel 2000 aa,bb,cc,"d , d", This works even in Excel 2000

aa,bb,cc, " d,d", This fails in Excel 2000 due to the space belore the 1st quote aa,bb,cc, " d ,d", This fails in Excel 2000 due to the space belore the 1st quote aa,bb,cc, " d , d", This fails in Excel 2000 due to the space belore the 1st quote

aa,bb,cc,"d,d " , This works even in Excel 2000 even with spaces before and after the 2nd quote. aa,bb,cc,"d ,d " , This works even in Excel 2000 even with spaces before and after the 2nd quote. aa,bb,cc,"d , d " , This works even in Excel 2000 even with spaces before and after the 2nd quote.

Rule: If you want to display a comma in a a cell (field) of a .csv file: "Start and end the field with a double quotes, but avoid white space before the 1st quote"

user_12475910
Related