IEnumerable becomes null while progressing in the code

Question

I have written a very simple windows application using .Net Core which produces an IEnumerable with 1000 records. In the following code which is shown as an example, my IEnumerable return the correct records while the second line of code return null.

IEnumerable myList = null; 
myList     = from item in myTable select ....();        
var result = myList.ToList();  //myList has 1000 records                  
int total  = myList.Count(); //myList is null

I don't understand why myList is not persisted and is changed to null. It looks like it is flushed after being used once.

Please see the real code below:

using var dbContextTransation = _context.Database.BeginTransaction();
try
{
    IEnumerable positionList = null; 
    var allPosition = csvReader.GetRecords();
    
    positionList =
        from location in allPosition
        select new Location()
        {                                              
            LocationDateTime = location.Fecha.Add(new TimeSpan( Convert.ToInt32( location.Hora.Left(2) ), Convert.ToInt32( location.Hora.Right(2) ), 0 ) ),                                               
            Latitude         = location.Latitud,
            Longitude        = location.Longitud
        };  
    
    var result = positionList.ToList();//List has 1000 records 
    int total = positionList.Count(); //List is null
        
    ....
    dbContextTransation.Commit();
//  _context.Dispose();
    Console.WriteLine("Committed " + DateTime.Now);
    
}
catch (Exception ex)
{
    dbContextTransation.Rollback();
    LogService.LogWrite(textFile, "ERR:" + ex.Message + " ->" + ex.InnerException.Message);                        
    return 0;
}

Thank you

Jon Koeter · Answer

You didn't mention you are using CsvHelper.

As stated here: https://joshclose.github.io/CsvHelper/getting-started/

"The GetRecords method will return an IEnumerable that will yield records. What this means is that only a single record is returned at a time as you iterate the records. That also means that only a small portion of the file is read into memory. Be careful though. If you do anything that executes a LINQ projection, such as calling .ToList(), the entire file will be read into memory. CsvReader is forward only, so if you want to run any LINQ queries against your data, you'll have to pull the whole file into memory. Just know that is what you're doing."

The CSVReader is using a StreamReader, which isn't reset between your two calls. Since you are already reading the entire file into a list by calling ToList, just use the result variable for whatever you are trying to do.

If you only want to know the count, and do nothing with the results, then calling Count() on your myList variable should work just fine.

If, for some reason, you still want to count the initial variable, you should reset your stream. This is also mentioned in the documentation of CsvHelper:

https://joshclose.github.io/CsvHelper/examples/prerequisites/streams/

Example:

using (var stream = new File.OpenRead("path\to\file"))
using (var reader = new StreamReader(stream))
{   
    // Read file content.
    var content = reader.ReadToEnd();

    // Go back to beginning of the stream.
    stream.Position = 0;

    // Read file content again.
    content = reader.ReadToEnd();
}

IEnumerable<> becomes null while progressing in the code

1 Answers