Hangfire - why is fire an forget Exception bubbling up

user_1135535 0
Viewed 234

I have a web api controller action that imports an ecommerce order. The calling server requires a response within 5 seconds or it considers it failed. As such, we want to accept the request and submit a fire and forget Hangfire job to run separately in it's own course and let the controller successfully return new HttpStatusCodeResult(HttpStatusCode.OK, msg); immediately, regardless of the result of that submitted job.

The problem is there was an Exception being thrown inside the method run in the enqueued Hangfire Job, i.e. it was thrown in the orderSyncAppServ.UpdateOrderFromWebhookAsync method, which is also an async method. We can see this in the Hangire dashboard looking at the job result and seeing the Exception.

Undesirably and unexpectingly, the calling server did not get the 200 response but rather received some error response. How could this be or where is my expectation misguided?

Update:

Of interest is that when I debug the code, the final return in the controller action is hit before a break point I have in the orderSyncAppServ.UpdateOrderFromWebhookAsync method...so now I'm really confused as to how the calling system is not getting the 200 response :( And that system is Shopify, not my own environment, and they don't provide a way to see the response they are getting, but my web server IIS logs do indeed show a 500 response code.

Here is my code:

[HttpPost]
public async Task<ActionResult> OrderUpdated (int storefrontId)
{
    string msg = "Successfully submitted update request to system.";
    string webhook = "orders/updated";

    try
    {
        var validationResult = await ValidateStorefrontWebhook(webhook, storefrontId);
        if (!validationResult.WasSuccessful) return new HttpStatusCodeResult(HttpStatusCode.OK, validationResult.Message);

        var orderSyncAppServ = new SyncErpWithPlacedOrdersTask();
        **Hangfire.BackgroundJob.Enqueue(() => orderSyncAppServ.UpdateOrderFromWebhookAsync(validationResult.Value, storefrontId));**
    }
    catch (Exception e)
    {
        msg = $"Exception webhook: {webhook} for storefront Id: {storefrontId}. {e.Message}.";
        _logger.Error(msg);
    }
    return new HttpStatusCodeResult(HttpStatusCode.OK, msg);
}

The elmah error thrown when calling the controller action:

System.ArgumentException: An item with the same key has already been added.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at System.Web.Mvc.JsonValueProviderFactory.AddToBackingStore(EntryLimitedDictionary backingStore, String prefix, Object value)
   at System.Web.Mvc.JsonValueProviderFactory.AddToBackingStore(EntryLimitedDictionary backingStore, String prefix, Object value)
   at System.Web.Mvc.JsonValueProviderFactory.AddToBackingStore(EntryLimitedDictionary backingStore, String prefix, Object value)
   at System.Web.Mvc.JsonValueProviderFactory.AddToBackingStore(EntryLimitedDictionary backingStore, String prefix, Object value)
   at System.Web.Mvc.JsonValueProviderFactory.AddToBackingStore(EntryLimitedDictionary backingStore, String prefix, Object value)
   at System.Web.Mvc.JsonValueProviderFactory.AddToBackingStore(EntryLimitedDictionary backingStore, String prefix, Object value)
   at System.Web.Mvc.JsonValueProviderFactory.AddToBackingStore(EntryLimitedDictionary backingStore, String prefix, Object value)
   at System.Web.Mvc.JsonValueProviderFactory.GetValueProvider(ControllerContext controllerContext)
   at System.Web.Mvc.ValueProviderFactoryCollection.GetValueProvider(ControllerContext controllerContext)
   at System.Web.Mvc.ControllerBase.get_ValueProvider()
   at System.Web.Mvc.ControllerActionInvoker.GetParameterValue(ControllerContext controllerContext, ParameterDescriptor parameterDescriptor)
   at System.Web.Mvc.ControllerActionInvoker.GetParameterValues(ControllerContext controllerContext, ActionDescriptor actionDescriptor)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_1.<BeginInvokeAction>b__0(AsyncCallback asyncCallback, Object asyncState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction(ControllerContext controllerContext, String actionName, AsyncCallback callback, Object state)
   at System.Web.Mvc.Controller.<>c.<BeginExecuteCore>b__152_0(AsyncCallback asyncCallback, Object asyncState, ExecuteCoreState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)
   at System.Web.Mvc.Controller.BeginExecuteCore(AsyncCallback callback, Object state)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)
   at System.Web.Mvc.Controller.BeginExecute(RequestContext requestContext, AsyncCallback callback, Object state)
   at System.Web.Mvc.MvcHandler.<>c.<BeginProcessRequest>b__20_0(AsyncCallback asyncCallback, Object asyncState, ProcessRequestState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)
   at System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.<>c__DisplayClass285_0.<ExecuteStepImpl>b__0()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

The Error shown in the hangfire job result: It was having an issue deserializing the JSON as a property was defined as not nullable long:

Newtonsoft.Json.JsonSerializationException
Error converting value {null} to type 'System.Int64'. Path '[0].line_items[0].variant_id', line 1, position 2584.

Newtonsoft.Json.JsonSerializationException: Error converting value {null} to type 'System.Int64'. Path '[0].line_items[0].variant_id', line 1, position 2584. ---> System.InvalidCastException: Null object cannot be converted to a value type.
   at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
   --- End of inner exception stack trace ---
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at ShopifyStorefrontAdapter.AppServ.ShopifyStoreFrontAdapter.<ConvertJsonOrderToOrderImportDTOList>d__13.MoveNext() in C:\inetpub\wwwroot\mozzo\Main\app\StoreFrontAdapters\ShopifyStorefrontAdapter\AppServ\ShopifyStoreFrontAdapter.cs:line 297
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at OTIS.AppServ.OrderMgmt.SyncErpWithPlacedOrdersTask.<UpdateOrderFromWebhook>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
hangfire async-await asp.net-mvc c#
1 Answers

Of interest is that when I debug the code, the final return in the controller action is hit before a break point I have in the orderSyncAppServ.UpdateOrderFromWebhookAsync method

This is expected behaviour. UpdateOrderFromWebhookAsync is performed out of process, with some delay, by Hangfire.

I think you may in fact have two errors :

  • the first one, Error converting value {null} to type 'System.Int64'... seems to be linked to validationResult.Value (null value for field variant_id of the first item where int64 is expected). This error is triggered when hangfire is processing level. This is not the one returned by the controller, which has returned long before.
  • the second one seems to be linked to the deserialization of your controller's parameter. This is the one the caller is getting, and this is why it does not get caught (it happens before the try...catch block). My advice on this matter : log the exact request received by your server. Reproduce locally. Track anomalies like duplicated parameters. Simplify the request until you identify the issue.

Sorry, not a real answer, but it was difficult to elaborate in comment.

user_12360440
Related