Azure Service Fabric Service Health Incorrectly Reported when Fabric URI is Reused Under New Application

Brandon Andersen 0
Viewed 26

Service Fabric Versions:

  • Runtime - 9.0.1028.9590

  • SDK - 6.0.1028.9590

When moving a Service Fabric service instance to a new Service Fabric Application instance of the same Application Type, we reuse the Fabric URI as it existed under the original application under which it was created. However, the new service instance is reporting its health status to the original application, rather than to the new application. Below is a synthetic example that replicates our issue as it appears in our development environments (both OneBox and an actual Service Fabric cluster):

Initial Conditions:

  • Application Types:
    1. A.B.Type
  • Applications:
    1. fabric:/A/B (hosting fabric:/A/B/C/D)
  • Services:
    1. fabric:/A/B/C/D (hosted under fabric:/A/B)

Process (Note: all steps are done either via PowerShell Service Fabric Module or Service Fabric Explorer, both produce identical results):

  1. Create an identical Application instance from the existing Application Type but with a modified name (fabric:/A/B/C)
  2. Remove the existing service (fabric:/A/B/C/D) from the originating application (fabric:/A/B)
  3. Recreate the service using the same name (fabric:/A/B/C/D) under the new application (fabric:/A/B/C)
  4. Observe the health reporting for the new service is reported back to the originating application (fabric:/A/B)

Final Conditions:

  • Application Types:
    1. A.B.Type
  • Applications:
    1. fabric:/A/B (hosting no services)
    2. fabric:/A/B/C (hosting fabric:/A/B/C/D)
  • Services:
    1. fabric:/A/B/C/D (hosted under fabric:/A/B/C, but service health and service/replica information reported to fabric:/A/B)

The recreated (new service with reused name) service knows it is being hosted under the new application:

Get-ServiceFabricServiceDescription -ServiceName 'fabric:/A/B/C/D'

ApplicationName              : fabric:/A/B/C
ServiceName                  : fabric:/A/B/C/D
ServiceTypeName              : Web1Type
ServiceKind                  : Stateful
HasPersistedState            : True
MinReplicaSetSize            : 3
TargetReplicaSetSize         : 3

The new service reports its own health correctly:

Get-ServiceFabricServiceHealth -ServiceName 'fabric:/A/B/C/D'

ServiceName           : fabric:/A/B/C/D
AggregatedHealthState : Ok
PartitionHealthStates :
                        PartitionId           : 6a2ecddb-0f26-4703-8141-0dcaf80faa67
                        AggregatedHealthState : Ok

HealthEvents          :
                        SourceId              : System.FM
                        Property              : State
                        HealthState           : Ok
                        SequenceNumber        : 504
                        SentAt                : 9/19/2022 20:41:39
                        ReceivedAt            : 9/19/2022 20:42:00
                        TTL                   : Infinite
                        Description           : Service has been created.
                        RemoveWhenExpired     : False
                        IsExpired             : False
                        HealthReportID        : FM_7.0_1008
                        Transitions           : Warning->Ok = 9/19/2022 20:42:00, LastError = 1/1/0001 00:00:00

HealthStatistics      :
                        Replica               : 3 Ok, 0 Warning, 0 Error
                        Partition             : 1 Ok, 0 Warning, 0 Error

The applications are not reporting the correct health (original application):

Get-ServiceFabricApplicationHealth -ApplicationName 'fabric:/A/B'

ApplicationName                 : fabric:/A/B
AggregatedHealthState           : Ok
ServiceHealthStates             :
                                  ServiceName           : fabric:/A/B/C/D
                                  AggregatedHealthState : Ok

DeployedApplicationHealthStates : None
HealthEvents                    :
                                  SourceId              : System.CM
                                  Property              : State
                                  HealthState           : Ok
                                  SequenceNumber        : 52266
                                  SentAt                : 9/19/2022 20:39:38
                                  ReceivedAt            : 9/19/2022 20:39:38
                                  TTL                   : Infinite
                                  Description           : Application has been created.
                                  RemoveWhenExpired     : False
                                  IsExpired             : False
                                  HealthReportID        : CM_7.0_1000
                                  Transitions           : Warning->Ok = 9/19/2022 20:39:38, LastError = 1/1/0001 00:00:00

HealthStatistics                :
                                  Replica               : 3 Ok, 0 Warning, 0 Error
                                  Partition             : 1 Ok, 0 Warning, 0 Error
                                  Service               : 1 Ok, 0 Warning, 0 Error
                                  DeployedApplication   : 0 Ok, 0 Warning, 0 Error
                                  DeployedServicePackage : 0 Ok, 0 Warning, 0 Error

New application:

Get-ServiceFabricApplicationHealth -ApplicationName 'fabric:/A/B/C'

ApplicationName                 : fabric:/A/B/C
AggregatedHealthState           : Ok
ServiceHealthStates             : None
DeployedApplicationHealthStates :
                                  ApplicationName       : fabric:/A/B/C
                                  NodeName              : _Node_0
                                  AggregatedHealthState : Ok

                                  ApplicationName       : fabric:/A/B/C
                                  NodeName              : _Node_3
                                  AggregatedHealthState : Ok

                                  ApplicationName       : fabric:/A/B/C
                                  NodeName              : _Node_1
                                  AggregatedHealthState : Ok

HealthEvents                    :
                                  SourceId              : System.CM
                                  Property              : State
                                  HealthState           : Ok
                                  SequenceNumber        : 52289
                                  SentAt                : 9/19/2022 20:41:26
                                  ReceivedAt            : 9/19/2022 20:41:26
                                  TTL                   : Infinite
                                  Description           : Application has been created.
                                  RemoveWhenExpired     : False
                                  IsExpired             : False
                                  HealthReportID        : CM_7.0_1000
                                  Transitions           : Warning->Ok = 9/19/2022 20:41:26, LastError = 1/1/0001 00:00:00

HealthStatistics                :
                                  Service               : 0 Ok, 0 Warning, 0 Error
                                  Replica               : 0 Ok, 0 Warning, 0 Error
                                  Partition             : 0 Ok, 0 Warning, 0 Error
                                  DeployedServicePackage : 3 Ok, 0 Warning, 0 Error
                                  DeployedApplication   : 3 Ok, 0 Warning, 0 Error

The service is actively reporting its health. Attempts to send updated health reports using either Send-ServiceFabricApplicationHealthReport or Send-ServiceFabricServiceHealthReport do not produce corrected results.

Creating subsequent new service instances under either application (without removing and recreating) result in correct health and service/replica reporting.

Question Is this a bug or is this scenario of reusing a service name that previously existed under another application not supported?

azure-service-fabric
0 Answers
Related