Microsoft Orleans in kubernetes StatefulSet POD crashes after several restart

user_325320 0
Viewed 271

Microsoft Orleans v3.4.3 Consul Clustering Running in K8S

siloBuilder
     .UseConsulClustering(opt =>
     {
         opt.Address = new Uri(AppConfig.Orleans.ConsulUrl);
         opt.AclClientToken = AppConfig.Orleans.AclClientToken;
     })
     .Configure<ClusterOptions>(options =>
     {
         options.ClusterId = AppConfig.Orleans.ClusterID;
         options.ServiceId = AppConfig.Orleans.ServiceID;
     })
     .siloBuilder.UseKubernetesHosting();

I configured the labels and environment variables for my POD accordingly to the doc.

          - name: ORLEANS_SERVICE_ID #Required by Orleans 
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['orleans/serviceId']
          - name: ORLEANS_CLUSTER_ID #Required by Orleans 
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['orleans/clusterId']
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['statefulset.kubernetes.io/pod-name']
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP

It is a StatefulSet with only 1 POD for testing. On initial startup, it works well. However, every time when I restart the POD, a new entry is created in Consul.

enter image description here

And It crashes in subsequent startup.

The log says

System.AggregateException: One or more errors occurred. (Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184])
 ---> Orleans.Runtime.MembershipService.OrleansClusterConnectivityCheckFailedException: Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184]
   at Orleans.Runtime.MembershipService.MembershipAgent.ValidateInitialConnectivity()
   at Orleans.Runtime.MembershipService.MembershipAgent.BecomeActive()
   at Orleans.Runtime.MembershipService.MembershipAgent.<>c__DisplayClass26_0.<<Orleans-ILifecycleParticipant<Orleans-Runtime-ISiloLifecycle>-Participate>g__OnBecomeActiveStart|6>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHost.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at UBS.OrleansServer.EntryPoint.Start() in /app/UBS/OrleansServer/EntryPoint.cs:line 102
   --- End of inner exception stack trace ---

I have to remove all the entries in Consul then restart the POD, then everything works fine.

The POD_NAME is the same for StatefulSet's POD, is it correct that each POD restart creates a new entry in Consul?

What could be the cause?

Thanks in advance

UPDATE After several rounds crashes and restart, finally it does not crash any more. And in log I see the following message

ProcessTableUpdate (called from DeclareDead) membership table: 5 silos, 1 are Active, 4 are Dead, Version=<31, 28123>. All silos: [SiloAddress=S10.18.123.244:11111:361163684 SiloName=ubs-job-dev-0 Status=Active, SiloAddress=S10.18.123.200:11111:361158057 SiloName=ubs-job-dev-0 Status=Dead, SiloAddress=S10.18.123.210:11111:361161905 SiloName=ubs-job-dev-0 Status=Dead, SiloAddress=S10.18.123.217:11111:361157424 SiloName=ubs-job-dev-0 Status=Dead, SiloAddress=S10.18.123.244:11111:361163558 SiloName=ubs-job-dev-0 Status=Dead]

The SiloName never changes and there is only one POD in StatefulSet, but it sees 5 silos, 4 of them are dead. It seems each new POD, even if pod name does not change, is seen as a new silo. Is that expected?

orleans consul
1 Answers
(Failed to get ping responses from 1 of 1 active silos. 
Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. 
Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184])

Looks like your membershiptable (in consul) thinks that you already have active silos in it. When your 'new' silo comes up and looks in the membershiptable it sees these active silos at the table's IP addresses.

To keep the cluster correct, a newly joining silo must be able to communicate with the existing silos. However if the membership table is incorrect (ip address with status 3/active) then you have a problem where the new silo is trying to ping the active silos and not being able to reach them will fail to join and fast itself.

You have a couple of solutions:

  • clear the consul table when deploying your solution
  • change the deploymentid on every deployment.

You obviously found the first solution (clear the table)

see silo lifecycle

user_384610
Related