Hangfire ContinueWithJob is stuck in awaiting state, though parent job has succeeded

user_809357 0
Viewed 1108

I have a few jobs executed one after the other via ContinueJobWith<MyHandler>(parentJobId, x => x.DoWork()).

However, the second job is not getting processed and always sits in Awaiting state: Parent is Succeeded, but job is not procssesed

The job itself is like this: Stuck in Enqueued

Why this can happen and where to check for resultion?

  • We are using Autofac as DI container, but we have our own JobActivator implementation because we have to deal with multitenancy.
  • We are using SQL Server 2019 for storage.
  • Hangfire version is 1.7.10
  • This is MVC 5 application
  • I've not seen any errors/exceptions in any logs or during debugging
  • After going through this I've added this to our Autofac registration
            builder.RegisterType<BackgroundJobStateChanger>()
                   .As<IBackgroundJobStateChanger>()
                   .InstancePerLifetimeScope();

This made no difference.

This is how the jobs are executed:

var parentJobId = _backgroundJobClient.Schedule<Handler>(h => h.ConvertCertToTraining(certId, command.SetUpOneToOneRelationship), TimeSpan.FromSeconds(1));
var filesCopyJObId = _backgroundJobClient.ContinueJobWith<Handler>(parentJobId, h => h.CopyAttachedFiles());
_backgroundJobClient.ContinueJobWith<Handler>(filesCopyJObId, h => h.NotifyUser(command.CertificationToBeConvertedIds, _principal.GetEmail()));

All the parameters are either int, bool or string. If I enqueue the awaiting jobs by hand, they are executed without issues.

I've added Hangfire logging, but could not see any issues there: server starts, stops, jobs change status, but could not see any obvious errors there.

What other things I should consider or where/how should I debug this?

hangfire c#
2 Answers

From the looks of it, the first job with ID 216348 completed successfully but your second job with ID 216349 is waiting on the parent ID of 216347. According to Hangfire documentation and experience, the parentID should be of the job that you are waiting to finish before executing the second job.

According to Hangfire documentation on ContinueJobWith, "Continuations are executed when its parent job has been finished". From your screenshots, it is not clear whats going on with JobID: 216347. Once this job, 216347 completes, job with ID 216349 should kick off. If you are expecting 216349 to start after 216348 finishes, check your code and make sure correct ParentID is passed to the second job.

Update

Based on this thread, add the ContinuationsSupportAttribute to GlobalJobFilters.Filter where you configure Hangfire service. This should make your Hangfire instance aware of continuation jobs.

GlobalJobFilters.Filters.Add(new ContinuationsSupportAttribute());
user_13905480

During the investigation, it turned out that we were replacing JobFilterProviderCollection with our own collection:

    var filterProviderCollection = new JobFilterProviderCollection
    {
        new MyFilterProvider(...)
    };
    var backgroundJobClient = new BackgroundJobClient(JobStorage.Current, filterProviderCollection);

MyFilterProvider looked like this:

    public IEnumerable<JobFilter> GetFilters(Job job)
    {
        return new JobFilter[]
        {
             new JobFilter(new HangfireTenantFilter(_tenantDetail, _principal), JobFilterScope.Global,  null),
             new JobFilter(new HangfireFunctionalityFilter(_functionalityFilter), JobFilterScope.Global, null),
        };
    }

It turned out that code that was doing work on Continuation only took filters from this filter collection and ContinuationsSupportAttribute was not executed there in the right time. So re-adding default Hangfire filters from GlobalJobFilters.Filters fixed the situation:

    public IEnumerable<JobFilter> GetFilters(Job job)
    {
        var customFilters = new List<JobFilter>()
        {
             new JobFilter(new HangfireTenantFilter(_tenantDetail, _principal), JobFilterScope.Global,  null),
             new JobFilter(new HangfireFunctionalityFilter(_functionalityFilter), JobFilterScope.Global, null),
        };
        customFilters.AddRange(GlobalJobFilters.Filters);

        return customFilters;
    }
user_8093570
Related