How to improve efficiency of loop

user_12991993 0
Viewed 100

I'm trying to save my data to csv file. When the rate is small(2,000/s to save), it works well. But when increase to 20,000/s, it works slowly.

class Channel 
{
    List<double> RawData { get; set; }
    ...
}

-----------------------------------

var channels = new List<Channel>();
// after fetch the data
var sw = new StreamWriter(FileStream, Encoding.Default);
for (i = 0; channels.First().RawData.Count; i ++)
{
    string line = DateTime.Now.ToString() + ",";
    line += string.Join(',', channels.Select(c => c.RawData[i]));
    sw.WriteLine(line);
    sw.Flush();
}

When the count of RawData of each channel reach to 20,000, the app will work slowly. Is there any solution to speed up the generation of line?

loops c#
2 Answers

There are a few things you can do to speed this up.

Typically, I/O operations are expensive (slowest) so try to avoid doing many of them, and instead write just once or a few times.

Amending strings is also slow when you do this many times. So instead you should use a StringBuilder.

And finally, it seems to me that the datetime you are using will in many cases (but maybe not all!) be the same value. If this is the case then you could call this just once and then output the same value.

You might also be able to speed up the channels Select statement (as per other posts).

So then you would have code such as this:

StringBuilder sb = new StringBuilder();
string dtValue = DateTime.Now.ToString();

for (int i = 0; channels.First().RawData.Count; i++)
{
    sb.Append(dtValue).Append(",").Append(channels.First().RawData[i]).Append(Environment.NewLine);    
}

//Write just once.
using (var sw = new StreamWriter(FileStream, Encoding.Default))
{
    sw.Writeline(sb.ToString());
    sw.Close();
}

Edit: Updated to use channels.First().RawData[i], as per comments.

user_8479620

You can use Write(string) method and do not concatenate strings.

Also StreamWriter.Flush method should not be called for each iteration.

for (i = 0; channels.First().RawData.Count; i ++) is not compiled and should not be used since string.Join(',', channels.Select(c => c.RawData[i])); will iterate over all items.

The sample code below does not do concatenation and does not read all data in the memory. It writes channel data to the stream for each channel separately. For sure it will take some time and the more data you have the longer it will work.

var channels = new List<Channel>();
// after fetch the data
using (var sw = new StreamWriter(FileStream, Encoding.Default)) // you have to check that both stream writer and `FileStream` instances are disposed properly 
{
foreach (var ch in channels)
{
sw.Write(string.Format("{0:ddMMyyyy hh:mm:ss},", DateTime.Now)); // it writes date time for every channel data 
foreach (var data in ch.RawData)
{
    sw.Write(string.Format("{0},", data.ToString(Culture.InvariantCulture)); // double.ToString() is culture specific so you can use Culture.InvariantCulture when converting double to string
}
sw.WriteLine(""); // last line break but note that all channel data are written in one line separated with commas
}
}

The main idea is not to concatenate all channels RawData list items to string. Definitely it will take additional time and memory to convert 20000 doubles to one string.

It just writes RawData items one by one to the file and saves time and memory (I hope)

user_9401820
Related