Long API Call - Async calls the answer?

user_2590004 0
Viewed 580

I am calling an external API which is slow. Currently if I havent called the API to get some orders for a while the call can be broken up into pages (pagingation).

So therefore fetching orders could be making multiple calls rather than the 1 call. Sometimes each call can be around 10 seconds per call so this could be about a minute in total which is far too long.

        GetOrdersCall getOrders = new GetOrdersCall();
        getOrders.DetailLevelList.Add(DetailLevelCodeType.ReturnSummary);
        getOrders.CreateTimeFrom = lastOrderDate;
        getOrders.CreateTimeTo = DateTime.Now;

        PaginationType paging = new PaginationType();
        paging.EntriesPerPage = 20;
        paging.PageNumber = 1;

        getOrders.Pagination = paging;

        getOrders.Execute();

        var response = getOrders.ApiResponse;
        OrderTypeCollection orders = new OrderTypeCollection();

        while (response != null && response.OrderArray.Count > 0)
        {
            eBayConverter.ConvertOrders(response.OrderArray, 1);

            if (response.HasMoreOrders)
            {
                getOrders.Pagination.PageNumber++;
                getOrders.Execute();

                response = getOrders.ApiResponse;
                orders.AddRange(response.OrderArray);
            }
        }

This is a summary of my code above... The getOrders.Execute() is when the api fires.

After the 1st "getOrders.Execute()" there is a Pagination result which tells me how many pages of data there are. My thinking is that I should be able to start an asnychronous call for each page and to populate the OrderTypeCollection. When all the calls are made and the collection is fully loaded then I will commit to the database.

I have never done Asynchronous calls via c# before and I can kind of follow Async await but I think my scenario falls out of the reading I have done so far?

Questions:

  1. I think I can set it up to fire off the multiple calls asynchronously but I'm not sure how to check when all tasks have been completed i.e. ready to commit to db.
  2. I've read somewhere that I want to avoid combining the API call and the db write to avoid locking in SQL server - Is this correct?

If someone can point me in the right direction - It would be greatly appreciated.

async-await c#
2 Answers

I think I can set it up to fire off the multiple calls asynchronously but I'm not sure how to check when all tasks have been completed i.e. ready to commit to db.

Yes you can break this up

The problem is ebay doesn't have an async Task Execute Method, so you are left with blocking threaded calls and no IO optimised async await pattern. If there were, you could take advantage of a TPL Dataflow pipeline which is async aware (and fun for the whole family to play), you could anyway, though i propose a vanilla TPL solution...

However, all is not lost, just fall back to Parallel.For and a ConcurrentBag<OrderType>

Example

var concurrentBag = new ConcurrentBag<OrderType>();


// make first call
// add results to concurrentBag
// pass the pageCount to the for
int pagesize = ...;

Parallel.For(1, pagesize,
   page =>
      {
         // Set up
         // add page
         // make Call 

         foreach(var order in getOrders.ApiResponse)
            concurrentBag.Add(order);
      });

// all orders have been downloaded
// save to db

Note : There are MaxDegreeOfParallelism which you configure, maybe set it to 50, though it wont really matter how much you give it, the Task Scheduler is not going to aggressively give you threads, maybe 10 or so initially and grow slowly.

The other way you can do this, is create your own Task Scheduler, or just spin up your own Threads with the old fashioned Thread Class

I've read somewhere that I want to avoid combining the API call and the db write to avoid locking in SQL server - Is this correct?

  • If you mean locking as in slow DB insert, use Sql Bulk Insert and update tools.
  • If you mean locking as in the the DB deadlock error message, then this is an entirely different thing, and worthy of its own question

Additional Resources

For(Int32, Int32, ParallelOptions, Action)

Executes a for (For in Visual Basic) loop in which iterations may run in parallel and loop options can be configured.

ParallelOptions Class

Stores options that configure the operation of methods on the Parallel class.

MaxDegreeOfParallelism

Gets or sets the maximum number of concurrent tasks enabled by this ParallelOptions instance.

ConcurrentBag Class

Represents a thread-safe, unordered collection of objects.

user_16129750

Yes ConcurrentBag<T> Class can be used to server the purpose of one of your questions which was: "I think I can set it up to fire off the multiple calls asynchronously but I'm not sure how to check when all tasks have been completed i.e. ready to commit to db."

This generic class can be used to Run your every task and wait all your tasks to be completed to do further processing. It is thread safe and useful for parallel processing.

user_16323050
Related