Optimized synchronization of access to a shared resource

user_8966726 0
Viewed 93

For my console application, I have implemented a simple file logger. The logger uses StringBuilder to keep appending log entries and writes the LogText data to the LogFile at the end. This results in having only one file I/O operation. The application execution must be extremely fast and hence, I have implemented Parallel.ForEach along with async-await and reduced I/O operations wherever possible.

The problem is that Logger is not thread safe. Using lock or Monitor to synchronize the shared resource logger inside Parallel.ForEach loop slows down the performance. Is there any optimal way to synchronize the shared resource which does not affect execution speed much?

I am open to alternative approaches or suggestions.

Logger.cs

public class Logger
{
    private readonly string LogFile;
    private readonly StringBuilder LogText;

    public Logger(string logFile)
    {
        LogFile = logFile;
        LogText = new StringBuilder();
    }

    public void Write(string message)
    {
        LogText.AppendLine(message);
    }
    
    public void WriteToFile()
    {
        File.AppendAllText(LogFile, LogText.ToString());
    }
}

Program.cs

public class Program
{
    public static void Main(string[] args)
    {
        string logFile = args[0];
        string workingDirectory = args[1];
        Logger logger = new Logger(logFile);
        logger.Write($"INFO | Execution Started");

        try
        {
            List<string> files = Directory.EnumerateFiles(workingDirectory, "*", SearchOption.AllDirectories).ToList();
            Parallel.ForEach(files, async file =>
            {
                List<string> results = await PerformCPUBoundComputationAsync();
                foreach(string result in results)
                {
                    logger.Write($"INFO | Item: {result}");
                }
                string response = await MakePostRequestAsync(results);
                logger.Write($"INFO | Response: {response}");
            });
        }
        catch (Exception ex)
        {
            logger.Write($"ERROR | {ex.Message}");
        }
        finally
        {
            logger.Write($"INFO | Execution Ended");
            logger.WriteToFile();
        }
    }
}
parallel-processing concurrency multithreading .net c#
1 Answers

This is not using your Logger class, I just want to show you an alternative approach

First we don't use Parallel.ForEach for IO bound work, it's not suitable, and we definitely don't give it an async lambda (which is an unobserved async void), which means Parallel.ForEach will finish before all the tasks complete.

As for your issues:

  1. To solve the completed task problem, let's use WhenAll
  2. To solve the thread safety issue, let's make a separate string builder for each task. It's a little allocatey, however it's lock free.
  3. Let's write all the logs at the end

The async and await pattern will return threads back to the thread pool when it's completing IO-bound work. The tasks scheduler will use those threads for CPU bound work.

var tasks = Directory
   .EnumerateFiles(workingDirectory, "*", SearchOption.AllDirectories)
   .Select(async x =>
   {
      var sb = new StringBuilder();
      List<string> results = // do cpu bound work, no need for fake async, 
      // thats to say no need to offload to another thread. 
      foreach (string result in results)
         sb.AppendLine($"INFO | Item: {result}");
      string response = await MakePostRequestAsync(results);
      sb.AppendLine($"INFO | Response: {response}");
      return sb;
   });

// await all your work to finish
var logs = await Task.WhenAll(tasks);

// write the results to the file
using var sw = new StreamWriter("FileName");
sw.WriteLine($"INFO | Execution Started");

foreach (var log in logs)
   sw.WriteLine(log);

sw.WriteLine($"INFO | Execution Ended");

Note : this may cause allocation and memory pressure depending on how big the log gets. In which case you may need to go back to a synchronization primitive, and take the penalty of the lock.

Another efficient approach, is to use something like Tpl Dataflow pipeline, do your calculations, do your posts, then batch the results for the writes which would likely be less allocatey and will have the advantage of a dealing with the sync and async workloads.

Processing (parallel)
       v
Posting (parallel)
       v
Batched Log Writes (Singular)
user_16129750
Related