How to append to the same csv file with ChoETL?

user_8397835 0
Viewed 360

Currently the code I have below calls JsonToCsv() to parse a JSON file and append to csv accordingly, so the result is as such:

result.csv

File Name    Page  Practice Name
file1.json   1     Associates & Co

However, as you can see i am using a for loop to iterate over a bunch of JSON files, and my intention is that it parses them and appends them to the csv. so the expected csv file should like like this if i have 3 json files:

File Name    Page  Practice Name
fileXYZ.json 1     XYZ & Co
fileAB2.json 1     ABC & Co
file1.json   1     Associates & Co

however, whats happening is that the csv file is getting overwritten and I only see the results from the last file in the csv file. How do i make it append to the same file?

static void Main(string[] args)
{
    //Output to CSV
    foreach (var jsonFile in Directory.GetFiles(jsonFilesPath))
    {
        JsonToCsv(jsonFile, csvFilePath);
    }
}

public static void JsonToCsv(string jsonInputFile, string csvFile)
{
    using (var p = new ChoJSONReader(jsonInputFile).WithJSONPath("$..readResults"))
    {
        using (var w = new ChoCSVWriter(csvFile)//.WithFirstLineHeader())
            .WithField("FileName", fieldName: "File Name")
            .WithField("Page")
            .WithField("PracticeName", fieldName: "Practice Name")
            .WithFirstLineHeader()
            )
        {
            // Limit the result to page 1 since the fields below only exist on the 1st page
            w.Write(p
                .Where(r1 => r1.page == 1)
                .Select(r1 =>
                {
                    var lines = (dynamic[])r1.lines;
                    return new
                    {
                        FileName = inputFile,
                        Page = r1.page,
                        PracticeName = lines[6].text,
                    };
                }
        ));
        }
    }
}
choetl c#
1 Answers

1st option. I'd recommend you to alter your method signature to accept multiple files.

public static void JsonToCsv(IEnumerable<string> jsonInputFiles, string csvFile)
{
    using (var w = new ChoCSVWriter(csvFile)
        .WithField("FileName", fieldName: "File Name")
        .WithField("Page")
        .WithField("PracticeName", fieldName: "Practice Name")
        .WithFirstLineHeader()
        )
    {
        foreach (var jsonInputFile in jsonInputFiles)
        {
            using (var p = new ChoJSONReader(jsonInputFile).WithJSONPath("$..readResults"))
            {
                w.Write(p
                    .Where(r1 => r1.page == 1)
                    .Select(r1 =>
                    {
                        var lines = (dynamic[])r1.lines;
                        return new
                        {
                            FileName = inputFile,
                            Page = r1.page,
                            PracticeName = lines[6].text,
                        };
                    }
                ));
            }
        }
    }
}

2nd option. Use FileStream with Append option, and do some extra code to handle different cases, see comments.

using (var fs = new FileStream(@"D:\file.csv", FileMode.Append, FileAccess.Write))
{
    using (var writer = new ChoCSVWriter(fs))
    {
        writer.WithField("FileName", fieldName: "File Name")
            .WithField("Page")
            .WithField("PracticeName", fieldName: "Practice Name");

        if (fs.Position == 0) // we don't need header if file already existed before
        {
            writer.WithFirstLineHeader();
        }

        writer.Write(new
        {
            FileName = "Test",
            Page = 1,
            PracticeName = "Abc"
        });

    }

    fs.Write(Environment.NewLine); // append new line carrier so we don't write to the same line when file reopened for writing
}

3rd option - from comments (based on 2nd). Switch between create and append options

bool someCondition = true;
FileMode fileMode = someCondition ? FileMode.Append : FileMode.Create;
using (var fs = new FileStream(@"D:\file.csv", fileMode, FileAccess.Write))
user_73130940
Related