orleanscontrib / orleans.clustering.kubernetes Goto Github PK

Orleans Membership provider for Kubernetes

License: MIT License

C# 100.00%

azure kubernetes kubernetes-cluster kubernetes-api orleans orleans-cluster distributed-systems distributed-computing dotnet-core orleans-framework

orleans.clustering.kubernetes's People

Contributors

Stargazers

Watchers

Forkers

philrowan-mtb jsukhabut applicativex pmsanford edwardsp jangocheng patrick-m-kendall felixti cotyar smitcham awesomedotnetcore alexmg liuqing19851022 ilchert fdbeirao xontab zeroshi lfzm teksoi rfrost-bps ryan-developer misha-fm kuzm1ch jianfengl nickcalarco joshuachou highlyunavailable galvesribeiro kirasq fr4nky80 rswilley avabin danpat04 docugami cdemi samember ronicarvalho shyamsundarb-arch vader1986 chandruchiku henric-root worldspawn los-tokenizadores clement128 games spatxos ibondy

orleans.clustering.kubernetes's Issues

Orleans v7 release Action failed

@galvesribeiro it seems that Nuget API key expired/is invalid so new release based on Orleans v7 cannot be pushed to NuGet. Could you have a look at that?

https://github.com/OrleansContrib/Orleans.Clustering.Kubernetes/actions/runs/3912727708/jobs/6687737885#step:5:32

1s
Run dotnet nuget push Orleans.Clustering.Kubernetes.*.nupkg -k *** -s https://api.nuget.org/v3/index.json
Pushing Orleans.Clustering.Kubernetes.7.0.0.nupkg to 'https://www.nuget.org/api/v2/package'...
  PUT https://www.nuget.org/api/v2/package/
  Forbidden https://www.nuget.org/api/v2/package/ 823ms
error: Response status code does not indicate success: 403 (The specified API key is invalid, has expired, or does not have permission to access the specified package.).

Thanks!

Cleaning of dead entries

Will you be including this feature? Orleans have introduced a dead entries cleaning in 2.3 as seen here dotnet/orleans#5389. Looks like @jcdickinson did some work on this in PR #19

KubeClient logs errors for non-error scenarios

Example

GetCustomObject logs an error if the object requested doesn't exist. When a silo starts up and attempts to insert a new row for itself, InsertRow checks to see if the silo object already exists by calling GetCustomObject. The "happy path" for a new silo spinning up is for when this custom object doesn't exist, so every new silo that spins up with throw an error.

On further inspection, it looks like the GetCustomObject case is a bug (see PR) but this is a problem with all the KubeClient methods. For example, when a silo starts up, it attempts to initialize the membership table. If it's not the first silo ever to start in the Kube cluster, the creation of those objects will fail with AlreadyExists. The end result of this is ~4 errors logged every time a new silo starts up in a normal way in an existing, healthy cluster.

Client cant connect when update orleans to 7.2.1

When i have this nuget in my client and run it, when it try to connect throw a exception.

System.MissingMethodException: 'Method not found: 'System.Threading.Tasks.Task`1<Orleans.GrainDirectory.AddressAndTag> Orleans.GrainDirectory.IDhtGrainDirectory.RegisterAsync(Orleans.Runtime.GrainAddress, Int32)'.'

I only try it on my local machine using UseLocalhostClustering not the kubernetes option and before the update works.

Anyone know how to resolve it? thanks

Configuration in Samples

I've been having issues getting the Orleans.Clustering.Kubernetes working in my k8s cluster. I've examined the samples but it is not clear how to properly configure this. To this end, I have three questions as follows:

Is there any significance in having unique ports per Orleans silo instance?
From what I can tell, there is no significance as each instance runs at a separate IP address. Further, the sample code always chooses ports 10025 and 20025 as the seed to Random is always '1':

   .ConfigureEndpoints(new Random(1).Next(10001, 10100), new Random(1).Next(20001, 20100))

In the sample code, there are two projects: one with just silo host and another silo host with the gateway enabled. From what I can tell from the samples, it is not clear the difference between these two. According to the Orleans documentation link, to disable the gateway the port should be set to zero. However, in the silo-only sample the gateway is set to new Random(1).Next(20001, 20100) which is effectively 20025. This can be seen in the following code (from KubeSiloHost):

private static async Task<ISiloHost> StartSilo()
{
    var builder = new SiloHostBuilder()
         .Configure<ClusterOptions>(options =>  { options.ClusterId = "testcluster"; options.ServiceId = "testservice"; })
        .ConfigureEndpoints(new Random(1).Next(10001, 10100), new Random(1).Next(20001, 20100))
        .UseKubeMembership(opt =>
        {
            //opt.APIEndpoint = "http://localhost:8001";
            //opt.CertificateData = "test";
            //opt.APIToken = "test";
            opt.CanCreateResources = true;
            opt.DropResourcesOnInit = true;
        })
        .AddMemoryGrainStorageAsDefault()
        .ConfigureApplicationParts(parts => parts.AddApplicationPart(typeof(HelloGrain).Assembly).WithReferences())
        .ConfigureLogging(logging => logging.AddConsole());

    var host = builder.Build();
    await host.StartAsync();
    return host;
}

What is the behavior of setting DropResourcesOnInit to true? Should this be set on all silo host instances? In examining the two silo host samples, KubeSiloHost has this set to true while KubeGatewayHost has this set to false.

Thank you for the assistance.

Minor feedback

Good stuff, dude! This will be extremely useful.
I've noticed that the options class is in the Orleans.Clustering.Kubernetes.Options namespace... I would just drop the Options sub-namespace, as this is one of the main surface APIs for the end-user.

Also no need to annotate it with Json attributes at all, since you are not deviating from the defaults anyway.

CertificateData option is not used by KubeClient

The KubeClusteringOptions.CertificateData option is not used by KubeClient preventing the use of a custom certificate.

.NET 8

The full .NET 8 Orleans is out, please update your package references.

ClusterId in KubeGatewayListProvider not used for Silo CRD filtering

ClusterId in https://github.com/OrleansContrib/Orleans.Clustering.Kubernetes/blob/fd688abe427de9962fb3a0c05da47898f077b4be/src/Orleans.Clustering.Kubernetes/KubeGatewayListProvider.cs is used just for logging. Those silos from seperate clusters trying to join other clusters. Is this expected behavior?

What are the steps for graceful silo update ?

Currently, example silo is deployed in separate k8s deployments. If something changes in application, I will need to redeploy silo from scratch, but k8s have already created silo resource in k8s and if new version of silo is deployed, pods are trying to connect to already dead silos and this causes exceptions.
What is the correct way to update silo applications ?

Deleting stale silos that are marked active but the pod has already been removed

After a pod exits prematurely it will create a silo entry with the status "Active". When this pod restarts it scans all silos from etcd and attempts to connect with its predecessor. I've noticed this behaviour several times when running in production on orleans 3 in a single silo cluster.

RBAC configuration

In the event someone is using RBAC within k8s, is there documentation on what access rights must be given to the Silos and Orleans clients so that they can interoperate using the k8s data store?

KubeMembershipTable Error: Initial ClusterVersionEntity entity doesn't exist.

Hello there,

I am testing the Kubernetes clustering on a minikube on windows 10.
RBAC is activated, and i deployer the CRDs :

Then I gave INSANE privileges to my pods :

And then I have this error :

The Silo won't start :-(

Any idea ?

Does not works with latest Orleans

Hi!
Looks like provider does not work with latest Orleans.
Any chances of updated release?

Sample docs say to create custom namespace but must use 'kubetest'

In the sample docs it says to select a custom namespace but later commands fail if the namespace isn't kubetest

Example:

Orleans.Clustering.Kubernetes/samples on  master on ☁️  (eu-west-2)
❯ kubectl apply -f ./Definitions/Silo-ServiceAccount.yaml --namespace orleans
clusterrole.rbac.authorization.k8s.io/orleanssilo unchanged
clusterrolebinding.rbac.authorization.k8s.io/orleanssilo unchanged
error: the namespace from the provided object "kubetest" does not match the namespace "orleans". You must pass '--namespace=kubetest' to perform this operation.

v7 fails with Microsoft.Orleans.Hosting.Kubernetes

When v7 is used with Microsoft.Orleans.Hosting.Kubernetes 7.0.0 it fails to interact with ClusterVersion CRD (see logs below). The hosting library is using KubernetesClient 7.0.7 while v7 clustering us using 6.0.19. The root cause is that KubernetesClient 7.0.7 library is used at runtime and it changes object returned by ListNamespacedCustomObjectAsync, GetNamespacedCustomObjectAsync, etc. from Newtonsoft.Json.Linq.JObject (assumed in 6.0.19) to System.Text.Json.JsonElement.

{"EventId":0,"LogLevel":"Warning","Category":"Orleans.Clustering.Kubernetes.KubeMembershipTable","Message":"We tried to Initialize ClusterVersion but fail. Ignoring for now...","Exception":"System.InvalidCastException: Unable to cast object of type \u0027System.Text.Json.JsonElement\u0027 to type \u0027Newtonsoft.Json.Linq.JObject\u0027.    at Orleans.Clustering.Kubernetes.KubeMembershipTable.TryInitClusterVersion()","State":{"Message":"We tried to Initialize ClusterVersion but fail. Ignoring for now...","{OriginalFormat}":"We tried to Initialize ClusterVersion but fail. Ignoring for now..."},"Scopes":[]}
{"EventId":0,"LogLevel":"Warning","Category":"Orleans.Clustering.Kubernetes.KubeMembershipTable","Message":"Failure reading all silo entries for cluster id actors-silo","Exception":"System.InvalidCastException: Unable to cast object of type \u0027System.Text.Json.JsonElement\u0027 to type \u0027Newtonsoft.Json.Linq.JObject\u0027.    at Orleans.Clustering.Kubernetes.KubeMembershipTable.GetClusterVersion()    at Orleans.Clustering.Kubernetes.KubeMembershipTable.ReadAll()","State":{"Message":"Failure reading all silo entries for cluster id actors-silo","ClusterId":"actors-silo","{OriginalFormat}":"Failure reading all silo entries for cluster id {ClusterId}"},"Scopes":[]}

Getting 'Forbidden' error while deploying to kuberenetes.

Hi,
Need help in deployment Orleans application in kubernetes using "Orleans.Clustering.Kubernetes" method.
and provided the Role and Rolebindings to the serviceAccount as below -

apiVersion: v1
kind: ServiceAccount
metadata:
name: orleanssilo
namespace: poc

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: orleanssilo
rules:
- apiGroups:
- orleans.dot.net
resources:
- silos
- clusterversions
verbs:
- get
- list
- watch
- create
- update
- patch
- delete

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: orleanssilo
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: orleanssilo
subjects:
- kind: ServiceAccount
name: orleanssilo
namespace: poc

But once deployed to kuberenetes facing the below issue -

fail: Orleans.Clustering.Kubernetes.KubeMembershipTable[0]
Initial ClusterVersionEntity entity doesn't exist.

Please suggest regarding the ClusterVersionEntity...

Unable to update Silo Entry

We have a large number of KubeMembershipTable exceptions coming from our orleans application, the errors start after the application has been running for 5-9 hours.

Message: Unable to find silo entry 172.19.16.62-50000-374433332.

CategoryName: Orleans.Clustering.Kubernetes.KubeMembershipTable

Exception type: System.InvalidOperationException

Failed method: Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext

Problem Id: System.InvalidOperationException at Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext

Assembly: Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null

Call Stack: System.InvalidOperationException:
at Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext (Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)

.NET 8.0 please

like in the topic

Pod crash on startup in StatefulSet with Consul Clustering

Version 3.4.3.

siloBuilder
     .UseConsulClustering(opt =>
     {
         opt.Address = new Uri(AppConfig.Orleans.ConsulUrl);
         opt.AclClientToken = AppConfig.Orleans.AclClientToken;
     })
     .UseKubernetesHosting();

I configured the labels and environment variables for my POD accordingly to the doc.

          - name: ORLEANS_SERVICE_ID #Required by Orleans 
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['orleans/serviceId']
          - name: ORLEANS_CLUSTER_ID #Required by Orleans 
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['orleans/clusterId']
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['statefulset.kubernetes.io/pod-name']
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP

Running Orleans in K8S StatefulSet, my CI tool deploys the K8S StatefulSet, and then it crashes on startup.

System.AggregateException: One or more errors occurred. (Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184])
 ---> Orleans.Runtime.MembershipService.OrleansClusterConnectivityCheckFailedException: Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184]
   at Orleans.Runtime.MembershipService.MembershipAgent.ValidateInitialConnectivity()
   at Orleans.Runtime.MembershipService.MembershipAgent.BecomeActive()
   at Orleans.Runtime.MembershipService.MembershipAgent.<>c__DisplayClass26_0.<<Orleans-ILifecycleParticipant<Orleans-Runtime-ISiloLifecycle>-Participate>g__OnBecomeActiveStart|6>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHost.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at UBS.OrleansServer.EntryPoint.Start() in /app/UBS/OrleansServer/EntryPoint.cs:line 102
   --- End of inner exception stack trace ---

System.MissingMethodException during ContainerCreating

Running into a strange issue when deploying to our kubernetes cluster:

Unhandled exception. System.MissingMethodException: Method not found: 'System.Threading.Tasks.Task`1<System.Object> k8s.KubernetesExtensions.CreateNamespacedCustomObjectAsync(k8s.IKubernetes, System.Object, System.String, System.String, System.String, System.String, System.String, System.Threading.CancellationToken)'.
at Orleans.Clustering.Kubernetes.KubeMembershipTable.TryInitClusterVersion()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Orleans.Clustering.Kubernetes.KubeMembershipTable.TryInitClusterVersion()
at Orleans.Clustering.Kubernetes.KubeMembershipTable.InitializeMembershipTable(Boolean tryInitTableVersion)
at Orleans.Runtime.MembershipService.MembershipTableManager.Start()
at Orleans.Runtime.MembershipService.MembershipTableManager.<>c__DisplayClass54_0.<<Orleans-ILifecycleParticipant-Participate>g__OnRuntimeGrainServicesStart|0>d.MoveNext()
--- End of stack trace from previous location ---
at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
at Orleans.Hosting.SiloHost.StartAsync(CancellationToken cancellationToken)
at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at POC.Silo.Program.Main(String[] args) in /home/vsts/work/1/s/POC.Silo/Program.cs:line 24
at POC.Silo.Program.
(String[] args)

Program.cs:line 24 represents the location where we're configuring our SiloHostBuilder:

...
siloBuilder
.ConfigureEndpoints(new Random(1).Next(10001, 10100), new Random(1).Next(20001, 20100))
.UseKubeMembership();
...

The recent change that I made was to upgrade the framework to net 5.0.

Membership failure

Hello

After upgrading to version 2.0.0 membership stopped working. I observe the following exceptions in log. There are no silo records in etcd.

System.ArgumentNullException: Value cannot be null. (Parameter 'source')
   at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)
   at System.Linq.Enumerable.Where[TSource](IEnumerable`1 source, Func`2 predicate)
   at Orleans.Clustering.Kubernetes.KubeMembershipTable.GetSilos()
   at Orleans.Clustering.Kubernetes.KubeMembershipTable.ReadAll()
   at Orleans.Runtime.MembershipService.MembershipTableManager.RefreshInternal(Boolean requireCleanup)
   at Orleans.Runtime.MembershipService.MembershipTableManager.Refresh()
   at Orleans.Runtime.MembershipService.MembershipTableManager.PeriodicallyRefreshMembershipTable()

The SSL connection could not be established

I deployed it on the local device by referring to https://github.com/OrleansContrib/Orleans.Clustering.Kubernetes/tree/master/samples, and then modified the dockerfile to sdk:8.0. The release was successful, and an error was reported when running.

[root@master samples]# kubectl get pods --namespace orleans-sample
NAME                               READY   STATUS             RESTARTS         AGE
orleans-client-564d574c44-bm42c    0/1     CrashLoopBackOff   29 (3m1s ago)    128m
orleans-gateway-76cb47958b-8762z   1/1     Running            18 (6m46s ago)   128m
orleans-gateway-76cb47958b-lvqvk   1/1     Running            18 (7m3s ago)    128m
orleans-silo-64b695766-56x5k       1/1     Running            18 (6m47s ago)   128m


[root@master samples]# kubectl logs orleans-client-564d574c44-bm42c -n  orleans-sample
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.
   at System.Net.Security.SslStream.SendAuthResetSignal(ReadOnlySpan`1 alert, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.ICustomObjectsOperations_ListNamespacedCustomObjectWithHttpMessagesAsync[T](String group, String version, String namespaceParameter, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.ListNamespacedCustomObjectWithHttpMessagesAsync(String group, String version, String namespaceParameter, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.CustomObjectsOperationsExtensions.ListNamespacedCustomObjectAsync(ICustomObjectsOperations operations, String group, String version, String namespaceParameter, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, CancellationToken cancellationToken)
   at Orleans.Clustering.Kubernetes.KubeGatewayListProvider.GetGateways() in /root/Orleans.Clustering.Kubernetes/src/Orleans.Clustering.Kubernetes/KubeGatewayListProvider.cs:line 48
   at Orleans.Messaging.GatewayManager.StartAsync(CancellationToken cancellationToken) in /_/src/Orleans.Core/Messaging/GatewayManager.cs:line 69
   at Orleans.OutsideRuntimeClient.<>c__DisplayClass46_0.<<StartInternal>b__0>d.MoveNext() in /_/src/Orleans.Core/Runtime/OutsideRuntimeClient.cs:line 154
--- End of stack trace from previous location ---
   at Orleans.OutsideRuntimeClient.<StartInternal>g__ExecuteWithRetries|46_3(Func`1 task, IClientConnectionRetryFilter retryFilter, CancellationToken cancellationToken) in /_/src/Orleans.Core/Runtime/OutsideRuntimeClient.cs:line 181
   at Orleans.OutsideRuntimeClient.StartInternal(CancellationToken cancellationToken) in /_/src/Orleans.Core/Runtime/OutsideRuntimeClient.cs:line 153
   at Orleans.OutsideRuntimeClient.Start(CancellationToken cancellationToken) in /_/src/Orleans.Core/Runtime/OutsideRuntimeClient.cs:line 142
   at Orleans.ClusterClient.StartAsync(CancellationToken cancellationToken) in /_/src/Orleans.Core/Core/ClusterClient.cs:line 72
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>g__LogAndRethrow|15_3(<>c__DisplayClass15_0&)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at KubeClient.Program.Main() in /root/Orleans.Clustering.Kubernetes/samples/KubeClient/Program.cs:line 34

And gateway and silo also reported similar errors.

warn: Orleans.Clustering.Kubernetes.KubeMembershipTable[0]
      Failure reading all silo entries for cluster id testcluster
      System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
       ---> System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.
         at System.Net.Security.SslStream.SendAuthResetSignal(ReadOnlySpan`1 alert, ExceptionDispatchInfo exception)
         at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
         at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
         at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
         at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
         at k8s.AbstractKubernetes.ICustomObjectsOperations_ListNamespacedCustomObjectWithHttpMessagesAsync[T](String group, String version, String namespaceParameter, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
         at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.ListNamespacedCustomObjectWithHttpMessagesAsync(String group, String version, String namespaceParameter, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
         at k8s.CustomObjectsOperationsExtensions.ListNamespacedCustomObjectAsync(ICustomObjectsOperations operations, String group, String version, String namespaceParameter, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, CancellationToken cancellationToken)
         at Orleans.Clustering.Kubernetes.KubeMembershipTable.GetClusterVersion() in /root/Orleans.Clustering.Kubernetes/src/Orleans.Clustering.Kubernetes/KubeMembershipTable.cs:line 422
         at Orleans.Clustering.Kubernetes.KubeMembershipTable.ReadAll() in /root/Orleans.Clustering.Kubernetes/src/Orleans.Clustering.Kubernetes/KubeMembershipTable.cs:line 149

I use kubeasz to install offline Kubernetes

hosts

# 'etcd' cluster should have odd member(s) (1,3,5,...)
[etcd]
192.168.110.14

# master node(s)
[kube_master]
192.168.110.14

# work node(s)
[kube_node]
192.168.110.14
192.168.110.61
192.168.110.120

# [optional] harbor server, a private docker registry
# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one
[harbor]
192.168.110.14 NEW_INSTALL=true

# [optional] loadbalance for accessing k8s from outside
[ex_lb]
#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443

# [optional] ntp server for the cluster
[chrony]
192.168.110.14

[all:vars]
# --------- Main Variables ---------------
# Secure port for apiservers
SECURE_PORT="6443"

# Cluster container-runtime supported: docker, containerd
CONTAINER_RUNTIME="containerd"

# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="calico"

# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"

# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.68.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="172.20.0.0/16"

# NodePort Range
NODE_PORT_RANGE="30000-32767"

# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="cluster.local"

# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/opt/kube/bin"

# Deploy Directory (kubeasz workspace)
base_dir="/etc/kubeasz"

# Directory for a specific cluster
cluster_dir="{{ base_dir }}/clusters/k8s"

# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

config.yml

############################
# prepare
############################
# 可选离线安装系统软件包 (offline|online)
INSTALL_SOURCE: "offline"

# 可选进行系统安全加固 github.com/dev-sec/ansible-collection-hardening
OS_HARDEN: false

# 设置时间源服务器【重要：集群内机器时间必须同步】
ntp_servers:
  - "192.168.110.14"
#  - "ntp1.aliyun.com"
#  - "time1.cloud.tencent.com"
#  - "0.cn.pool.ntp.org"

# 设置允许内部时间同步的网络段，比如"10.0.0.0/8"，默认全部允许
local_network: "0.0.0.0/0"


############################
# role:deploy
############################
# default: ca will expire in 100 years
# default: certs issued by the ca will expire in 50 years
CA_EXPIRY: "876000h"
CERT_EXPIRY: "438000h"

# kubeconfig 配置参数
CLUSTER_NAME: "cluster1"
CONTEXT_NAME: "context-{{ CLUSTER_NAME }}"

# k8s version
K8S_VER: "1.23.1"

############################
# role:etcd
############################
# 设置不同的wal目录，可以避免磁盘io竞争，提高性能
ETCD_DATA_DIR: "/var/lib/etcd"
ETCD_WAL_DIR: ""


############################
# role:runtime [containerd,docker]
############################
# ------------------------------------------- containerd
# [.]启用容器仓库镜像
ENABLE_MIRROR_REGISTRY: false

# [containerd]基础容器镜像
SANDBOX_IMAGE: "easzlab/pause:3.6"

# [containerd]容器持久化存储目录
CONTAINERD_STORAGE_DIR: "/var/lib/containerd"

# ------------------------------------------- docker
# [docker]容器存储目录
DOCKER_STORAGE_DIR: "/var/lib/docker"

# [docker]开启Restful API
ENABLE_REMOTE_API: false

# [docker]信任的HTTP仓库
INSECURE_REG: '["127.0.0.1/8"]'


############################
# role:kube-master
############################
# k8s 集群 master 节点证书配置，可以添加多个ip和域名（比如增加公网ip和域名）
MASTER_CERT_HOSTS:
  - "192.168.110.14"
  #- "www.test.com"

# node 节点上 pod 网段掩码长度（决定每个节点最多能分配的pod ip地址）
# 如果flannel 使用 --kube-subnet-mgr 参数，那么它将读取该设置为每个节点分配pod网段
# https://github.com/coreos/flannel/issues/847
NODE_CIDR_LEN: 24


############################
# role:kube-node
############################
# Kubelet 根目录
KUBELET_ROOT_DIR: "/var/lib/kubelet"

# node节点最大pod 数
MAX_PODS: 110

# 配置为kube组件（kubelet,kube-proxy,dockerd等）预留的资源量
# 数值设置详见templates/kubelet-config.yaml.j2
KUBE_RESERVED_ENABLED: "no"

# k8s 官方不建议草率开启 system-reserved, 除非你基于长期监控，了解系统的资源占用状况；
# 并且随着系统运行时间，需要适当增加资源预留，数值设置详见templates/kubelet-config.yaml.j2
# 系统预留设置基于 4c/8g 虚机，最小化安装系统服务，如果使用高性能物理机可以适当增加预留
# 另外，集群安装时候apiserver等资源占用会短时较大，建议至少预留1g内存
SYS_RESERVED_ENABLED: "no"

# haproxy balance mode
BALANCE_ALG: "roundrobin"


############################
# role:network [flannel,calico,cilium,kube-ovn,kube-router]
############################
# ------------------------------------------- flannel
# [flannel]设置flannel 后端"host-gw","vxlan"等
FLANNEL_BACKEND: "vxlan"
DIRECT_ROUTING: false

# [flannel] flanneld_image: "quay.io/coreos/flannel:v0.10.0-amd64"
flannelVer: "v0.15.1"
flanneld_image: "easzlab/flannel:{{ flannelVer }}"

# [flannel]离线镜像tar包
flannel_offline: "flannel_{{ flannelVer }}.tar"

# ------------------------------------------- calico
# [calico]设置 CALICO_IPV4POOL_IPIP=“off”,可以提高网络性能，条件限制详见 docs/setup/calico.md
CALICO_IPV4POOL_IPIP: "Always"

# [calico]设置 calico-node使用的host IP，bgp邻居通过该地址建立，可手工指定也可以自动发现
IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}"

# [calico]设置calico 网络 backend: brid, vxlan, none
CALICO_NETWORKING_BACKEND: "brid"

# [calico]更新支持calico 版本: [v3.3.x] [v3.4.x] [v3.8.x] [v3.15.x]
calico_ver: "v3.19.3"

# [calico]calico 主版本
calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}"

# [calico]离线镜像tar包
calico_offline: "calico_{{ calico_ver }}.tar"

# ------------------------------------------- cilium
# [cilium]CILIUM_ETCD_OPERATOR 创建的 etcd 集群节点数 1,3,5,7...
ETCD_CLUSTER_SIZE: 1

# [cilium]镜像版本
cilium_ver: "v1.4.1"

# [cilium]离线镜像tar包
cilium_offline: "cilium_{{ cilium_ver }}.tar"

# ------------------------------------------- kube-ovn
# [kube-ovn]选择 OVN DB and OVN Control Plane 节点，默认为第一个master节点
OVN_DB_NODE: "{{ groups['kube_master'][0] }}"

# [kube-ovn]离线镜像tar包
kube_ovn_ver: "v1.5.3"
kube_ovn_offline: "kube_ovn_{{ kube_ovn_ver }}.tar"

# ------------------------------------------- kube-router
# [kube-router]公有云上存在限制，一般需要始终开启 ipinip；自有环境可以设置为 "subnet"
OVERLAY_TYPE: "full"

# [kube-router]NetworkPolicy 支持开关
FIREWALL_ENABLE: "true"

# [kube-router]kube-router 镜像版本
kube_router_ver: "v0.3.1"
busybox_ver: "1.28.4"

# [kube-router]kube-router 离线镜像tar包
kuberouter_offline: "kube-router_{{ kube_router_ver }}.tar"
busybox_offline: "busybox_{{ busybox_ver }}.tar"


############################
# role:cluster-addon
############################
# coredns 自动安装
dns_install: "yes"
corednsVer: "1.8.6"
ENABLE_LOCAL_DNS_CACHE: true
dnsNodeCacheVer: "1.21.1"
# 设置 local dns cache 地址
LOCAL_DNS_CACHE: "169.254.20.10"

# metric server 自动安装
metricsserver_install: "yes"
metricsVer: "v0.5.2"

# dashboard 自动安装
dashboard_install: "yes"
dashboardVer: "v2.4.0"
dashboardMetricsScraperVer: "v1.0.7"

# ingress 自动安装
ingress_install: "no"
ingress_backend: "traefik"
traefik_chart_ver: "10.3.0"

# prometheus 自动安装
prom_install: "no"
prom_namespace: "monitor"
prom_chart_ver: "12.10.6"

# nfs-provisioner 自动安装
nfs_provisioner_install: "no"
nfs_provisioner_namespace: "kube-system"
nfs_provisioner_ver: "v4.0.2"
nfs_storage_class: "managed-nfs-storage"
nfs_server: "192.168.110.14"
nfs_path: "/data/nfs"

############################
# role:harbor
############################
# harbor version，完整版本号
HARBOR_VER: "v2.1.3"
HARBOR_DOMAIN: "k8s.harbor.com"
HARBOR_TLS_PORT: 8443

# if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down'
HARBOR_SELF_SIGNED_CERT: true

# install extra component
HARBOR_WITH_NOTARY: false
HARBOR_WITH_TRIVY: false
HARBOR_WITH_CLAIR: false
HARBOR_WITH_CHARTMUSEUM: true

Orleans v7

Hi, is there any plan to add support for .net 7 (Orleans v7)?

I could start working on it as well.

Tomas

Versioning on CRDs

As we plan to enhance CRDs with features and to avoid breaking changes to actually break things, we should use version fields of all CRDs inside the codebase to control at which version some features would be enabled or not.

An example of some enhancements I envision are:

Automatic health report/check
Placement constraints + heterogeneous clusters
Metadata

Auto-expose Gateway port

The provider should have an option to automagically expose the Gateway/Proxy port when a silo has the Gateway installed (i.e. when siloEntry.ProxyPort > 0). That would open the Orleans cluster to have Orleans clients outside Kubernetes Cluster boundaries allowing non-containerized apps to talk to the cluster (very useful in on-premises scenarios).

That would (optionally) remove the need for people to expose the Gateway port to outside Kubernetes cluster.

However, there is an issue to consider while doing that. If the port is fixed, each Kubernetes worker node would not be able to run more than 1 silo otherwise, we would have port conflicts.

To workaround that, we could (optionally) randomly generate port numbers for the Gateway and creating respective NodePort objects but, that would make on-premises deployments a potential nightmare in terms of firewalls and routing since the ports are unknown until the silo is initialized. In cloud services like AKS, the Azure firewall is integrated with Kube API and it (optionally) open ports automatically whenever a new service is exposed by a pod.

Need to investigate that and come up with a solution.

Feature request: allow field selector on status

For housekeeping it would be neat if one could write:

kubectl get silos --all-namespaces --field-selector "status=dead"

But this currently returns the error Error from server (BadRequest): Unable to find "silos" that match label selector "", field selector "status=dead": field label not supported: status

See more here: https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/

Complete example of deployment please?

My silo is up and running on Kubernetes:

> kubectl get services
NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
contact-trace-silo   NodePort    10.98.120.212   <none>        2777:30403/TCP,30000:31986/TCP   26m

But my client deployed to another pod can't seem to connect to it:

...
Found 1 knownGateways from Gateway listProvider [gwy.tcp://127.0.0.1:30000/324611186]
...
Connection attempt to endpoint S127.0.0.1:30000:0 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 127.0.0.1:30000. Error: ConnectionRefused

The thing that confuses me is that it should not be using localhost, it should be using the pod ip address 10.98.120.212, since each pod has it's own address. And the ports get mapped differently also? Shouldn't the KubeMembershipTable be taking care of this?

The following contains a simple repro:
tutorial.zip

Different Applications colliding

I've followed your samples and have set up an application on our kube cluster using the following [default] settings:

silo:

return siloHost
	.ConfigureEndpoints(
		new Random(1).Next(30001, 30100),
		new Random(1).Next(20001, 20100),
		listenOnAnyHostAddress: true
	)
	.UseKubeMembership(opt =>
	{
		opt.CanCreateResources = true;
		//opt.DropResourcesOnInit = true;
});

client:

clientBuilder.UseKubeGatewayListProvider();

This works fine. However, when I added the second application things started getting weird. The apps started behaving differnetly with every request, sometimes working, sometimes throwing:

Unexpected: Cannot find an implementation class for grain interface 1039391738

and sometimes throwing: Cannot find an implementation class for grain interface: Odin.Gaming.Contracts.Cache.IGameCacheWorkerGrain

The apps comprise each of 4 pods with a web api client and 8 pods for the silos. I suspect that somehow the apps are conflicting.

Is this some misconfiguration on my part?
Thanks.

Silo's fail in sequence if I go from 2 to 3 instances

Hi, we've been using this library for a while in a dev context and we just tried to scale beyond 2 instances for the first time.

What happens is almost immediately one silo will fail, it then start up and another will fail. If we swap to azure storage membership this doesn't happen.

Here's a silo in its last moments

[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6fdaeb9d InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #457[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6fdaeb9d to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@81befabc InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #458[ForwardCount=1], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@81befabc, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4b5895209cfe93c5086f48ac81befabc0000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@81befabc InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #458[ForwardCount=1] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@81befabc to (null) after Non-existent activation. Attempt 1
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #458[ForwardCount=3], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @486061fd394345e11cdc7fc99abde0990000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #458[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[1m�[33mwarn�[39m�[22m�[49m: Orleans.Messaging[101032]
[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #458[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@9abde099 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@db71f98e InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategy #459[ForwardCount=1], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@db71f98e, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4a48f9d36f860ff5fe58c944db71f98e0000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@db71f98e InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategy #459[ForwardCount=1] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@db71f98e to (null) after Non-existent activation. Attempt 1
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategy #459[ForwardCount=3], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4c9d690e9939a1bd086666daa0a232860000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategy #459[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[1m�[33mwarn�[39m�[22m�[49m: Orleans.Messaging[101032]
[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategy #459[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@a0a23286 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@aecb5a9b InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategies #460[ForwardCount=1], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@aecb5a9b, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @44c2de53fd88c5212b34325eaecb5a9b0000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@aecb5a9b InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategies #460[ForwardCount=1] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@aecb5a9b to (null) after Non-existent activation. Attempt 1
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategies #460[ForwardCount=3], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4dc092a0f1b53586476b33b7f41c3bb90000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategies #460[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[1m�[33mwarn�[39m�[22m�[49m: Orleans.Messaging[101032]
[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetCompatibilityStrategies #460[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@f41c3bb9 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@4659d4a0 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #461[ForwardCount=1], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@4659d4a0, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @49715c7e2bf900a0148980974659d4a00000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@4659d4a0 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #461[ForwardCount=1] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@4659d4a0 to (null) after Non-existent activation. Attempt 1
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #461[ForwardCount=3], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4d60b6f2dee90cc52fb699646b714a960000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #461[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[1m�[33mwarn�[39m�[22m�[49m: Orleans.Messaging[101032]
[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategy #461[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@6b714a96 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@19e561a8 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #462[ForwardCount=1], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@19e561a8, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4860c1ca5d2dcb5de91afc3119e561a80000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@19e561a8 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #462[ForwardCount=1] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@19e561a8 to (null) after Non-existent activation. Attempt 1
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Runtime.Dispatcher[100147]
[projects-silo-695f465bb5-n7s6m silo]        Intermediate NonExistentActivation for message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #462[ForwardCount=3], with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4, Full GrainId: [GrainId: *grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo-0x5F7A4DC2, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 1601850818 (0x5F7A4DC2), KeyExtension: projects-silo], Full ActivationId: @4253032b40b2e8be1b667e48c75d25a40000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
[projects-silo-695f465bb5-n7s6m silo]           at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[32minfo�[39m�[22m�[49m: Orleans.Messaging[101031]
[projects-silo-695f465bb5-n7s6m silo]        Trying to forward Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #462[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  �[40m�[1m�[33mwarn�[39m�[22m�[49m: Orleans.Messaging[101032]
[projects-silo-695f465bb5-n7s6m silo]        Failed to forward message Request S10.244.6.207:10025:359102100*stg/TypeManager/00000011@S00000011->S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4 InvokeMethodRequest Orleans.Runtime.Versions.IVersionStoreGrain:GetSelectorStrategies #462[ForwardCount=3] from S10.244.6.207:10025:359102100*grn/Orleans.Runtime.Versions.VersionStoreGrain/0+projects-silo@c75d25a4 to (null) after Non-existent activation. Attempt 3
[projects-silo-695f465bb5-n7s6m silo]  Process terminated. FATAL EXCEPTION from Orleans.Runtime.MembershipService.MembershipTableManager. Context: I have been told I am dead, so this silo will stop! I should be Dead according to membership table (in CleanupTableEntries): entry = [SiloAddress=S10.244.6.207:10025:359102100 SiloName=projects-silo-695f465bb5-n7s6m Status=Dead HostName=projects-silo-695f465bb5-n7s6m ProxyPort=20025 RoleName= UpdateZone=0 FaultZone=0 StartTime = 2021-05-19 06:35:00.796 GMT IAmAliveTime = 2021-05-19 06:35:04.823 GMT Suspecters = [S10.244.6.210:10025:359102156] SuspectTimes = [2021-05-19 06:35:57.959 GMT]].. Exception: null.
[projects-silo-695f465bb5-n7s6m silo]  Current stack:    at System.Environment.get_StackTrace()
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.FatalErrorHandler.OnFatalException(Object sender, String context, Exception exception)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.KillMyselfLocally(String reason)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.CleanupMyTableEntries(MembershipTableData table)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.CleanupMyTableEntries(MembershipTableData table)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.RefreshInternal(Boolean requireCleanup)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Clustering.Kubernetes.KubeMembershipTable.ReadAll()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Clustering.Kubernetes.KubeMembershipTable.GetSilos()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.KubernetesExtensions.ListNamespacedCustomObjectAsync(IKubernetes operations, String group, String version, String namespaceParameter, String plural, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.Kubernetes.ListNamespacedCustomObjectWithHttpMessagesAsync(String group, String version, String namespaceParameter, String plural, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.WatcherDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ThreadPoolWorkQueue.Dispatch()
[projects-silo-695f465bb5-n7s6m silo]     at System.Environment.FailFast(System.String, System.Exception)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.FatalErrorHandler.OnFatalException(System.Object, System.String, System.Exception)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.KillMyselfLocally(System.String)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager+<CleanupMyTableEntries>d__47.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Orleans.Runtime.MembershipService.MembershipTableManager+<CleanupMyTableEntries>d__47, Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]](<CleanupMyTableEntries>d__47 ByRef)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Orleans.Runtime.MembershipService.MembershipTableManager+<CleanupMyTableEntries>d__47, Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]](<CleanupMyTableEntries>d__47 ByRef)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager.CleanupMyTableEntries(Orleans.MembershipTableData)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Runtime.MembershipService.MembershipTableManager+<RefreshInternal>d__34.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Runtime.MembershipService.MembershipTableManager+<RefreshInternal>d__34, Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Runtime.MembershipService.MembershipTableManager+<RefreshInternal>d__34, Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Runtime.MembershipService.MembershipTableManager+<RefreshInternal>d__34, Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Clustering.Kubernetes.KubeMembershipTable+<ReadAll>d__9.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<ReadAll>d__9, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<ReadAll>d__9, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<ReadAll>d__9, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at Orleans.Clustering.Kubernetes.KubeMembershipTable+<GetSilos>d__17.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<GetSilos>d__17, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<GetSilos>d__17, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Orleans.Clustering.Kubernetes.KubeMembershipTable+<GetSilos>d__17, Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.KubernetesExtensions+<ListNamespacedCustomObjectAsync>d__1581.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.KubernetesExtensions+<ListNamespacedCustomObjectAsync>d__1581, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.KubernetesExtensions+<ListNamespacedCustomObjectAsync>d__1581, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.KubernetesExtensions+<ListNamespacedCustomObjectAsync>d__1581, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.Kubernetes+<ListNamespacedCustomObjectWithHttpMessagesAsync>d__818.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.Kubernetes+<ListNamespacedCustomObjectWithHttpMessagesAsync>d__818, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.Kubernetes+<ListNamespacedCustomObjectWithHttpMessagesAsync>d__818, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.Kubernetes+<ListNamespacedCustomObjectWithHttpMessagesAsync>d__818, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpClient+<FinishSendAsyncBuffered>d__70.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpClient+<FinishSendAsyncBuffered>d__70, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at k8s.WatcherDelegatingHandler+<SendAsync>d__0.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.WatcherDelegatingHandler+<SendAsync>d__0, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].ExecutionContextCallback(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.WatcherDelegatingHandler+<SendAsync>d__0, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[k8s.WatcherDelegatingHandler+<SendAsync>d__0, KubernetesClient, Version=2.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122de]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.DiagnosticsHandler+<SendAsync>d__2.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.DiagnosticsHandler+<SendAsync>d__2, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.RedirectHandler+<SendAsync>d__4, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d__47.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<SendWithRetryAsync>d__47, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnectionPool+<SendWithNtConnectionAuthAsync>d__48.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<SendWithNtConnectionAuthAsync>d__48, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.__Canon)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Http.HttpConnection+<SendAsyncCore>d__53.MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnection+<SendAsyncCore>d__53, System.Net.Http, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Security.SslStream+<ReadAsyncInternal>d__214`1[[System.Net.Security.SslStream+SslReadAsync, System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReadAsyncInternal>d__214`1[[System.Net.Security.SslStream+SslReadAsync, System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task.RunContinuations(System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Security.SslStream+<<FillBufferAsync>g__InternalFillBufferAsync|215_0>d`1[[System.Net.Security.SslStream+SslReadAsync, System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[projects-silo-695f465bb5-n7s6m silo]     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<<FillBufferAsync>g__InternalFillBufferAsync|215_0>d`1[[System.Net.Security.SslStream+SslReadAsync, System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
[projects-silo-695f465bb5-n7s6m silo]     at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean)
[projects-silo-695f465bb5-n7s6m silo]     at System.Threading.ThreadPoolWorkQueue.Dispatch()

Client using out of K8S

If the client is running out of K8S, how to initialize it?

var client = new ClientBuilder()
        ...
        .UseKubeGatewayListProvider() // Optionally use the configure delegate to specify the namespace where you cluster is running.
        ...
        .Build();

Seems the client must run in the same K8S as well.

Directory.RegisterAsync S10.100.0.133:11111:286399169*cli/693b40b9@9aa135a9 failed.

Hey, I'm getting a weird issue:

[19:19:43:723 WRN] UnregisterManyAsync 1 failed.
System.InvalidOperationException: Grain directory is stopping
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable`1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary`2& forward, List`1 tasks, String context)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterManyAsync(List`1 addresses, UnregistrationCause cause, Int32 hopCount)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.Catalog.FinishDestroyActivations(List`1 list, Int32 number, MultiTaskCompletionSource tcs)
[19:19:43:724 WRN] UnregisterManyAsync 3 failed.
System.InvalidOperationException: Grain directory is stopping
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable`1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary`2& forward, List`1 tasks, String context)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterManyAsync(List`1 addresses, UnregistrationCause cause, Int32 hopCount)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.Catalog.FinishDestroyActivations(List`1 list, Int32 number, MultiTaskCompletionSource tcs)
[19:19:44:527 ERR] Directory.RegisterAsync S10.100.0.133:11111:286399169*cli/693b40b9@9aa135a9 failed.
System.InvalidOperationException: Grain directory is stopping
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19:19:44:528 ERR] OnClientRefreshTimer has thrown an exceptions.
System.InvalidOperationException: Grain directory is stopping
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnClientRefreshTimer(Object data)
[19:19:44:560 ERR] RunClientMessagePump has thrown exception
System.OperationCanceledException: The operation was canceled.
   at System.Collections.Concurrent.BlockingCollection`1.TryTakeWithNoTimeValidation(T& item, Int32 millisecondsTimeout, CancellationToken cancellationToken, CancellationTokenSource combinedTokenSource)
   at System.Collections.Concurrent.BlockingCollection`1.TryTake(T& item, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Collections.Concurrent.BlockingCollection`1.Take(CancellationToken cancellationToken)
   at Orleans.Runtime.HostedClient.RunClientMessagePump()
[19:19:44:582 WRN] Process is exiting

Not really sure what the issue could be, sample app works fine. This is an existing Orleans app i'm trying to convert to run on top of kubernetes cluster. crd's are there, I can see it creating custom resources.

address: 10.100.0.133
apiVersion: orleans.dot.net/v1
clusterId: clarityorleansclusterid
generation: 286399169
hostname: cli-784b77474b-w2wsr
iAmAliveTime: "2019-01-28T19:19:43.8210054+00:00"
kind: OrleansSilo
metadata:
  name: 10.100.0.133-11111-286399169
  namespace: orleans
port: 11111
proxyPort: 30000
siloName: Silo_63485
startTime: "2019-01-28T19:19:30.0103034+00:00"
status: dead
suspectingSilos:
- 10.100.0.133:11111@286399169
suspectingTimes:
- 2019-01-28 19:19:43.821 GMT

apiVersion: orleans.dot.net/v1
clusterId: clarityorleansclusterid
clusterVersion: 4
kind: OrleansClusterVersion
metadata:
  name: clarityorleansclusterid
  namespace: orleans

Any pointers?

ISiloBuilder and Orleans 2.3.0 HostBuilder Extensions

I upgraded to the latest Orleans 2.3.0 build today in hopes of using the new HostBuilder extensions. However, I noticed the HostBuilder.UseOrleans method expects an action with a parameter of ISiloBuilder instead of ISiloHostBuilder that Orleans.Clustering expects.

The change to support this looks to be a straightforward addition to the extensions class where the this parameter is of ISiloBuilder type.

        public static ISiloBuilder UseKubeMembership(this ISiloBuilder builder,
            Action<KubeClusteringOptions> configureOptions)
        {
            return builder.ConfigureServices(services => services.UseKubeMembership(configureOptions));
        }

The readme contains an invalid link to cluster membership protocol documention.

Orleans Extended Cluster Membership Protocol. link points to a non-existent page in the Orleans repository documentation from the following

A page exists that defines cluster management here:

https://dotnet.github.io/orleans/Documentation/implementation/cluster_management.html

Fix: Orleans.Hosting.KubernetesHosting: System.MissingMethodException

Multiple Kubernetes client updates have changed the method signatures and when using a newer version of the client (e.g. when using KubeOps in conjunction with Orleans in the same app) the following error is thrown:

Unhandled exception. System.MissingMethodException: Method not found: 'System.Threading.Tasks.Task`1<System.Object> k8s.KubernetesExtensions.CreateNamespacedCustomObjectAsync(k8s.IKubernetes, System.Object, System.String, System.String, System.String, System.String, System.String, System.Threading.CancellationToken)'.
    at Orleans.Clustering.Kubernetes.KubeMembershipTable.TryInitClusterVersion()
    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
    at Orleans.Clustering.Kubernetes.KubeMembershipTable.TryInitClusterVersion()
    at Orleans.Clustering.Kubernetes.KubeMembershipTable.InitializeMembershipTable(Boolean tryInitTableVersion)
    at Orleans.Runtime.MembershipService.MembershipTableManager.Start()

The fix is to upgrade the Kubernetes C# SDK to the latest version, and I have a PR in with the Orleans repo (dotnet/orleans#7346) to do that as well as a PR ready for this repo once I see the disposition of their fix timing but both Orleans.Hosting.Kubernetes and Orleans.Clustering.Kubernetes should to be upgraded to match versions and move in lockstep so one doesn't drag the other down into an incompatible state.

Unable to get gateways from Kube objects for cluster 'Test'

I deployed my project using the instructions in the 'sample' directory, but i keep getting the following error:

fail: Orleans.Clustering.Kubernetes.KubeGatewayListProvider[0] Unable to get gateways from Kube objects for cluster Test System.Net.Http.HttpRequestException: Connection timed out (10.96.0.1:443) ---> System.Net.Sockets.SocketException (110): Connection timed out at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) at k8s.Kubernetes.<>c.<<CreateHttpClient>b__820_0>d.MoveNext() --- End of stack trace from previous location --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at k8s.Kubernetes.ListNamespacedCustomObjectWithHttpMessagesAsync(String group, String version, String namespaceParameter, String plural, Nullable1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable1 limit, String resourceVersion, String resourceVersionMatch, Nullable1 timeoutSeconds, Nullable1 watch, Nullable1 pretty, Dictionary2 customHeaders, CancellationToken cancellationToken)
at k8s.KubernetesExtensions.ListNamespacedCustomObjectAsync(IKubernetes operations, String group, String version, String namespaceParameter, String plural, Nullable1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable1 limit, String resourceVersion, String resourceVersionMatch, Nullable1 timeoutSeconds, Nullable1 watch, Nullable1 pretty, CancellationToken cancellationToken) at Orleans.Clustering.Kubernetes.KubeGatewayListProvider.GetGateways() fail: Orleans.Messaging.GatewayManager[100921] Exception occurred during RefreshSnapshotLiveGateways_TimerCallback -> listProvider.GetGateways() System.Net.Http.HttpRequestException: Connection timed out (10.96.0.1:443) ---> System.Net.Sockets.SocketException (110): Connection timed out at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) at k8s.Kubernetes.<>c.<<CreateHttpClient>b__820_0>d.MoveNext() --- End of stack trace from previous location --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at k8s.Kubernetes.ListNamespacedCustomObjectWithHttpMessagesAsync(String group, String version, String namespaceParameter, String plural, Nullable1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable1 limit, String resourceVersion, String resourceVersionMatch, Nullable1 timeoutSeconds, Nullable1 watch, Nullable1 pretty, Dictionary2 customHeaders, CancellationToken cancellationToken)
at k8s.KubernetesExtensions.ListNamespacedCustomObjectAsync(IKubernetes operations, String group, String version, String namespaceParameter, String plural, Nullable1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable1 limit, String resourceVersion, String resourceVersionMatch, Nullable1 timeoutSeconds, Nullable1 watch, Nullable1 pretty, CancellationToken cancellationToken) at Orleans.Clustering.Kubernetes.KubeGatewayListProvider.GetGateways() at Orleans.Messaging.GatewayManager.RefreshSnapshotLiveGateways_TimerCallback(Object context) in /_/src/Orleans.Core/Messaging/GatewayManager.cs:line 257

Got "Could not load type 'k8s.Models.V1ObjectMeta' from assembly KubernetesClient" exception if using in FSharp Orleans Project

Hi there,

I was just tried to use version 2.1.0 in my fsharp project with orleans 3.7, but I got Could not load type 'k8s.Models.V1ObjectMeta' from assembly KubernetesClient exception when the application startup. Does anyone encountered similar issue? thanks

[02:05:40 ERR] Lifecycle start canceled due to errors at stage 8000: System.TypeLoadException: Could not load type 'k8s.Models.V1ObjectMeta' from assembly 'KubernetesClient, Version=11.0.0.0, Culture=neutral, PublicKeyToken=null'.
   at Orleans.Clustering.Kubernetes.KubeMembershipTable.TryInitClusterVersion()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Orleans.Clustering.Kubernetes.KubeMembershipTable.TryInitClusterVersion()
   at Orleans.Clustering.Kubernetes.KubeMembershipTable.InitializeMembershipTable(Boolean tryInitTableVersion)
   at Orleans.Runtime.MembershipService.MembershipTableManager.Start() in /_/src/Orleans.Runtime/MembershipService/MembershipTableManager.cs:line 164
   at Orleans.Runtime.MembershipService.MembershipTableManager.<>c__DisplayClass54_0.<<Orleans-ILifecycleParticipant<Orleans-Runtime-ISiloLifecycle>-Participate>g__OnRuntimeGrainServicesStart|0>d.MoveNext() in /_/src/Orleans.Runtime/MembershipService/MembershipTableManager.cs:line 859

Implement sidecar support

In kubernetes there is a pattern for Sidecard containers. They essentially live in the same pod as the main container.

A valid use case for sidecar containers in Orleans context, is that you may want to use the main container as a ASP.Net Core container which expose a WebAPI and have a sidecar container which is the silo. As kubernetes runtime thinks that a pod is alive and healthy only if all containers within that pod are alive, in this particular case, the WebAPI has a guarantee that the silo it is talking is alive.

With this feature, I would like to be able to configure the gateway list provider to instead of querying the Kube API for the list of silos, to always think it has a single gateway, which is on localhost as all container in the same pod share the same networking namespace.

That would avoid the need to call Kube API and hence all the configuration necessary to do so from the client side, leaving that for the silo/sidecar container.

Orleans kubectl extension?

Kubernetes default client, kubectl, introduced on 1.8+ release the ability to create plugins which extend kubectl capabilities as described here https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/.

Would be nice to have kubectl orleans XXX where XXX is a command to do something with the cluster. That way, we would abstract some of the Kubernetes complicated management operations from Orleans users.

To create that, a new Orleans.Clustering.Kubernetes.Extensions self-contained .Net Core console app would be added which could be deployed in multiple ways (Chocolatey, Docker container, apt-get, you name it) and installed as a kubectl plugin.

The suggested initial available commands would be:

--scale-silo=X: Scale the silos to a particular X number of replicas.
--info=counter: Get some information based on an exposed Orleans counter, for example, Activation Count, etc.
--cluster-info: Get the current membership records

Invoke grains would be possible, but would require having the interface assemblies...

Would like to hear from others if that is useful, which other commands should be added etc...

Thanks

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.