Service Fabric - Enable read on statefull secondary replicas

Question

Service Fabric - Enable read on statefull secondary replicas

Yves Royer 1

Hi there,

We're tweaking the operations, especially read operation, on our SF cluster.
Some context: we have 1 stateful service which persists data that only changes once or twice a year. This service/data is partitioned in 8 partitions so at this level we already have spread requests accross the cluster. At this moment when fetching data we only use the primary replica per partition and we're trying to enable reads on secondary replicas as well but without success.

At listener side we changed our implementation from this

protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
{
        return this.CreateServiceRemotingReplicaListeners();
}

to this:

protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
{
    if (!_args.ListenOnSecondary)
    {
        // use the default and preferred way of constructing the listeners
        return this.CreateServiceRemotingReplicaListeners();
    }

    var settings = new FabricTransportRemotingListenerSettings
    {
        UseWrappedMessage = true
    };

    return new[]
    {
        new ServiceReplicaListener(context => new FabricTransportServiceRemotingListener(context, this, settings),
                                   _args.ListenerName,
                                   _args.ListenOnSecondary)
    };
}

note: "_args" is just a class containing a couple of properties which is injected in the ctor of the base class including the setting "ListenOnSecondary" to enable this at the registered listener. Nothing special otherwise ;)

The consumer on it's turn calls the code

ServiceProxy.Create<T>(ServiceUri, new ServicePartitionKey(partitionId), replicaSelector)

ServiceUri => constructed based service name
partitionId => calculated based on the way our partitioning is set up for this service
replicaSelector => in this case set to "TargetReplicaSelector.RandomReplica"

When executing on our cluster we sometimes get this error (when a secondadary replica is used):

The primary or stateless instance for the partition '3509a0e4-ba3e-4a58-bc53-a25a73e7b7de' has invalid address, this means that right address from the replica/instance is not registered in the system.

Remark: when using the "default" (TargetReplicaSelector.Default) replica selector everything works fine!

Any thought or things we're missing?

1 answer

Your answer

Answer 1

Hello Yves !

Thank you for posting on Microsoft Learn Q&A.

I think the client is connecting with the default listener name while the service you registered is a named so when the resolver picks a secondary, the address for that listener name isn’t found and the primary works with default because you happen to hit the endpoint that exists.

You need to open a remoting listener on both primary and secondaries with a known endpoint name that also exists in the ServiceManifest.

// ServiceManifest.xml
// <Endpoint Name="remotingListener" Protocol="tcp" Port="0" />
protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
{
    var settings = new FabricTransportRemotingListenerSettings
    {
        UseWrappedMessage = true
    };
    return new[]
    {
        new ServiceReplicaListener(
            ctx => new FabricTransportServiceRemotingListener(ctx, this, settings),
            name: "remotingListener",          // MUST match client
            listenOnSecondary: true)           // <- key bit
    };
}

In the client you need to pecify the same listener name and target secondary replicas only :

// Make sure client and service both use the same remoting stack/settings
var proxyFactory = new ServiceProxyFactory(c =>
    new FabricTransportServiceRemotingClientFactory(
        remotingClientFactorySettings: new FabricTransportRemotingSettings
        {
            UseWrappedMessage = true
        }));
var proxy = proxyFactory.CreateServiceProxy<IMyService>(
    serviceUri: ServiceUri,
    partitionKey: new ServicePartitionKey(partitionId),
    targetReplicaSelector: TargetReplicaSelector.RandomSecondaryReplica, // or RandomReplica
    listenerName: "remotingListener");                                   // MUST match service

Share via

Service Fabric - Enable read on statefull secondary replicas

1 answer

Your answer