I have a desktop application where users can select a number of rows displayed in a form (customs declarations actually) and send them via web service to get a response with the information of their current state, that needs to be updated in the database.
Since now this process was made synchronously. Get the first row, consume the web service, wait for the response, update the state in database and then move to the next row. But the number of rows processed each time is growing (4000-5000 at a time) and its taking a while to complete.
So what I did is parallelize the consumption of the web service for each row more or less like this:
'I can't parallelize the transactions with the database (business logic reasons),
'so I use a queue for storing the responses as they arrive
Private ResponseQueue as ConcurrentQueue(Of WebServiceResponseDataType)
Private NumberOfRowsToSend as Integer
Private CounterResponsesProcessed as Integer
Public Sub Send(ByVal dataTableRowsToSend As DataTable)
NumberOfRowsToSend = dataTableRowstoSend.Rows.Count()
'I use a separate Thread to launch the method that will be updating the database
'as the queue is filled with responses
Dim responseProcessorThread as Thread
responseProcessorThread = New Thread(AddressOf ResponseProcessor)
responseProcessorThread.Start()
For Each row In dataTableRowsToSend.Rows
Dim dataToSend() as Object
'Do some previous validations and data formatting here.
'I need to get some IDs from another tables in the database
'and increment them before consuming the web servcie.
'To avoid duplicate IDs I keep this part synchronous.
'If the validations are passed I use the ThreadPool to queue the consumption of the WebService
ThreadPool.QueueUserWorkItem(New WaitCallback(AddressOf WebServiceConsumption)), dataToSend)
Next
'Wait for all responses to be processed
'PS. I'm controlling if there is an exception during the webservice consumption,
'generating a fake response in that case so the user can see what went wrong
responseProcessorThread.Join()
End Sub
Private Sub ResponseProcessor()
Dim WebServiceResponse As WebServiceResponseDataType
While CounterResponsesProcessed < NumberOfRowsToSend
If ResponseQueue.Count > 0 AndAlso ResponseQueue.TryDequeue(WebServiceResponse) Then
UpdateRowStateInDataBase(WebServiceResponse)
CounterResponsesProcessed += 1
End If
End While
End Sub
Private Sub WebServiceConsumption (ByVal dataToSend() as Object)
'This function consumes the web service and waits for the response.
'When the response arrives it goes to the queue:
ResponseQueue.Enqueue(responseData)
End Sub
With this code the process is much faster since I don't have to wait for the response of the previous row to start with the next one.
But I'm not used to parallel programming and I'm not sure if this is a good approach or if I did something considered bad practice.
I.e. I dont feel comfortable using a counter to control when all the responses are updated in the database to know when the process is finished. Also after writing this code I have been reading that I should avoid using Threads directly and use Tasks instead.