C# Parallel - Duplicate iteration

user_2513695 0
Viewed 116

I am facing strange issue where files in iteration are being called twice.

Parallel.ForEach(context.files.Where(x => x.in_queue == true && x.Uploaded == false), new ParallelOptions { MaxDegreeOfParallelism = 5 }, (fl, state) =>
                {
                    //File upload code
                        fl.in_queue = false;
                        fl.Uploaded = true;
                        log.Debug(string.Format("{0} uploaded", fl.FileName));
                });

I am seeing same files more than once in the log, but the context doesn't contain any duplicate entries.

.net c#
1 Answers

NUnit test that shows issue is not reproducable using plain code.

Like @RBarryYoung points to, a concurrency issue is most likely the problem.

public class CustomFile
{
    public bool in_queue { get; set; }
    public bool Uploaded { get; set; }
    public string FileName { get; set; }
}

[Test]
public void ParallelTest()
{
    var files = Enumerable.Range(1, 10000)
        .Select(it =>
            new CustomFile {in_queue = true, Uploaded = false, FileName = $"{it}_my_file.txt"}
        );

    var usedFileNames = new ConcurrentBag<string>();

    Parallel.ForEach(files
            .Where(x => x.in_queue == true && x.Uploaded == false),
        new ParallelOptions {MaxDegreeOfParallelism = 5}, (fl) =>
        {
            fl.in_queue = false;
            fl.Uploaded = true;
            usedFileNames.Add(fl.FileName);
        });

    Assert.IsFalse(usedFileNames.GroupBy(x => x).Any(x => x.Count() > 1));
}
user_140724980
Related