Is is possible to optimimize this piece of code with ConcurrentExclusiveSchedulerPair?

user_16774174 0
Viewed 47

The problem is currently it runs pretty fine with limited number of reports (around 10 000), but on environments with much more reports it fails with Big amount of simultaneous requests for server causing heavy load on the server:

private void PerformSearch(SearchReportModel item)
{
    var tasks = new List<Task>();

    foreach (var term in item.SearchTerms)
    {
        var model = GetBaseQueryModel(item.Site);
        tasks.Add(Task.Factory.StartNew(() => CheckSearchTerm(model, term, item.Site, item.Language)));
    }
    try
    {
        Task.WaitAll(tasks.ToArray());
    }            
    catch (Exception ex)
    {
        Log4NetLogger.LogError(ex, ex.ToString());
        throw;
    }
}
task-parallel-library c#
1 Answers

Assuming that CheckSearchTerm performs API calls in question I would suggest just using SemaphoreSlim to throttle the load:

Represents a lightweight alternative to Semaphore that limits the number of threads that can access a resource or pool of resources concurrently.

var tasks = new List<Task>();
var maximumRequests = 100; // maximum simultaneous invocations 
var limiter = new SemaphoreSlim(maximumRequests); // possibly make a global one via static variable or move inside of CheckSearchTerm
foreach (var term in item.SearchTerms)
{
    var model = GetBaseQueryModel(item.Site);
    tasks.Add(Task.Run(async () =>
    {
        await limiter.WaitAsync();
        try
        {
            CheckSearchTerm(model, term, item.Site, item.Language);
        }
        finally
        {
            limiter.Release();
        }
        
    }));
}

Another option is using Parallel.ForEach/PLINQ with ParallelOptions.MaxDegreeOfParallelism/WithDegreeOfParallelism set to required value.

user_25012790
Related