CA Ingress doesn't resolve about lf-k8s-hlf-webinar

[Neekoy]

The CA pod logs say a whole lot of the following:

$ kubectl logs nginx-ingress-controller-6f6f69f94f-mnjzh -n ingress-controller
W0206 08:48:37.920117       7 controller.go:1071] Error getting SSL certificate "blockchain/ca--tls": local SSL certificate blockchain/ca--tls was not found. Using default certificate
W0206 08:48:41.252825       7 controller.go:826] Service "blockchain/ca-hlf-ca" does not have any active Endpoint.
W0206 08:48:41.252996       7 controller.go:1071] Error getting SSL certificate "blockchain/ca--tls": local SSL certificate blockchain/ca--tls was not found. Using default certificate
W0206 08:48:44.586118       7 controller.go:826] Service "blockchain/ca-hlf-ca" does not have any active Endpoint.

The logs of the CA are the following:

$ kubectl logs ca-hlf-ca-798c98c84f-brflq -n blockchain
2019/02/07 08:26:55 [DEBUG] Received request for /cainfo
2019/02/07 08:26:55 [INFO] 10.0.0.230:38032 GET /cainfo 200 0 "OK"
2019/02/07 08:26:59 [DEBUG] Received request for /cainfo
2019/02/07 08:26:59 [INFO] 10.0.0.230:38070 GET /cainfo 200 0 "OK"

Also yes, I did change the helm_values/ca_values.yaml to the following:

image:
  tag: 1.2.0

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: "letsencrypt-production"
  path: /
  hosts:
    # TODO: Change this back
    - orbix-fabric-ca-3.orbixpay.com
  tls:
    - secretName: ca--tls
      hosts:
        # TODO: Change this back
        - orbix-fabric-ca-3.orbixpay.com

Not sure what else to try and thanks for your help. :)

[Neekoy]

Btw I figured out the part with the Ingress - I set up an AWS ALB Ingress Controller and now it's working (that was a little bit tricky but I figured it out in the end). Now I'm getting new errors though - the CA Authority is displaying the following:

2019/02/11 14:27:59 [DEBUG] Received request for /cainfo
2019/02/11 14:27:59 [DEBUG] Initializing DB
2019/02/11 14:27:59 [DEBUG] Initializing 'postgres' database at 'host=ca-postgresql port=5432 user=**** password=**** dbname=fabric_ca sslmode=disable'
2019/02/11 14:27:59 [DEBUG] Using postgres database, connecting to database...
2019/02/11 14:27:59 [DEBUG] Database Name: fabric_ca
2019/02/11 14:27:59 [DEBUG] Connecting to PostgreSQL server, using connection string: host=ca-postgresql port=5432 user=**** password=**** dbname=fabric_ca sslmode=disable
2019/02/11 14:27:59 [WARNING] Failed to connect to database 'fabric_ca'
2019/02/11 14:27:59 [DEBUG] Connecting to PostgreSQL server, using connection string: host=ca-postgresql port=5432 user=**** password=**** dbname=postgres sslmode=disable
2019/02/11 14:27:59 [WARNING] Failed to connect to database 'postgres'
2019/02/11 14:27:59 [DEBUG] Connecting to PostgreSQL server, using connection string: host=ca-postgresql port=5432 user=**** password=**** dbname=template1 sslmode=disable
2019/02/11 14:27:59 [WARNING] Failed to connect to database 'template1'
2019/02/11 14:27:59 [INFO] 10.0.0.230:50940 GET /cainfo 500 0 "cainfo handler failed to initialize DB: Failed to create user registry for PostgreSQL: Failed to connect to Postgres database. Postgres requires connecting to a specific database, the following databases were tried: [fabric_ca postgres template1]. Please create one of these database before continuing"

Can you confirm if this is something you can possibly look into? Really not sure why it wouldn't be able to connect with the default credentials :/

[alexvicegrab]

Yes, most likely reason is that you deployed the PostgreSQL on an existing PVC, and this means it can't connect to it and has different credentials saved (likely on disk). Delete deployment, then destroy related PVCs and reinstall

[Neekoy]

Awesome, I'll give that a try tomorrow. As of the Ingress part, I'll post a comment with how it works on AWS EKS with the ALB Ingress Controller in case someone stumbles upon this question.

Thanks a lot for the help, Alex. :)

[Neekoy]

I actually found out how to install it and now the setup is going properly. Super awesome, thanks a lot guys and I'll ask if something goes on again.

[Neekoy]

Alright I resolved another thingie, but now I'm getting the following under Create Channel:

kubectl exec -n blockchain $PEER_POD -- peer channel create -o ord1-hlf-ord.blockchain.svc.cluster.local:7050 -c mychannel -f /hl_config/channel/mychannel.tx


2019-02-12 15:08:31.444 UTC [msp] setupSigningIdentity -> DEBU 032 Signing identity expires at 2020-02-12 12:59:00 +0000 UTC
2019-02-12 15:08:31.444 UTC [msp] Validate -> DEBU 033 MSP AidTechMSP validating identity
2019-02-12 15:08:31.444 UTC [msp] GetDefaultSigningIdentity -> DEBU 034 Obtaining default signing identity
2019-02-12 15:08:31.444 UTC [grpc] Printf -> DEBU 035 parsed scheme: ""
2019-02-12 15:08:31.444 UTC [grpc] Printf -> DEBU 036 scheme "" not registered, fallback to default scheme
2019-02-12 15:08:31.444 UTC [grpc] Printf -> DEBU 037 ccResolverWrapper: sending new addresses to cc: [{ord1-hlf-ord.blockchain.svc.cluster.local:7050 0  <nil>}]
2019-02-12 15:08:31.444 UTC [grpc] Printf -> DEBU 038 ClientConn switching balancer to "pick_first"
2019-02-12 15:08:31.444 UTC [grpc] Printf -> DEBU 039 pickfirstBalancer: HandleSubConnStateChange: 0xc42021a950, CONNECTING
2019-02-12 15:08:31.447 UTC [grpc] Printf -> DEBU 03a pickfirstBalancer: HandleSubConnStateChange: 0xc42021a950, READY
2019-02-12 15:08:31.447 UTC [channelCmd] InitCmdFactory -> INFO 03b Endorser and orderer connections initialized
2019-02-12 15:08:31.447 UTC [msp] GetDefaultSigningIdentity -> DEBU 03c Obtaining default signing identity
2019-02-12 15:08:31.447 UTC [msp] GetDefaultSigningIdentity -> DEBU 03d Obtaining default signing identity
2019-02-12 15:08:31.447 UTC [msp/identity] Sign -> DEBU 03e Sign: plaintext: 0ACF070A0A416964546563684D535012...120E0A0C4D79436F6E736F727469756D 
2019-02-12 15:08:31.447 UTC [msp/identity] Sign -> DEBU 03f Sign: digest: 02B4DFEDE014CC5C4CF37B578C01BEC1FC858D2C1955C67FC1D5CA879D52966C 
2019-02-12 15:08:31.447 UTC [msp] GetDefaultSigningIdentity -> DEBU 040 Obtaining default signing identity
2019-02-12 15:08:31.447 UTC [msp] GetDefaultSigningIdentity -> DEBU 041 Obtaining default signing identity
2019-02-12 15:08:31.447 UTC [msp/identity] Sign -> DEBU 042 Sign: plaintext: 0A86080A1508021A0608EFC28BE30522...BABA619C9D3CF38E46A8EAF604357C2C 
2019-02-12 15:08:31.447 UTC [msp/identity] Sign -> DEBU 043 Sign: digest: 58E5298E5F628C5439624EA6D82A7CEB208825842F020588E9F52D4265581EC0 
2019-02-12 15:08:31.447 UTC [grpc] Printf -> DEBU 044 parsed scheme: ""
2019-02-12 15:08:31.447 UTC [grpc] Printf -> DEBU 045 scheme "" not registered, fallback to default scheme
2019-02-12 15:08:31.447 UTC [grpc] Printf -> DEBU 046 ccResolverWrapper: sending new addresses to cc: [{ord1-hlf-ord.blockchain.svc.cluster.local:7050 0  <nil>}]
2019-02-12 15:08:31.447 UTC [grpc] Printf -> DEBU 047 ClientConn switching balancer to "pick_first"
2019-02-12 15:08:31.447 UTC [grpc] Printf -> DEBU 048 pickfirstBalancer: HandleSubConnStateChange: 0xc42016eb60, CONNECTING
2019-02-12 15:08:31.449 UTC [grpc] Printf -> DEBU 049 pickfirstBalancer: HandleSubConnStateChange: 0xc42016eb60, READY
Error: got unexpected status: SERVICE_UNAVAILABLE -- will not enqueue, consenter for this channel hasn't started yet
command terminated with exit code 1

The first row is the command that I entered.

Not sure what this means either :(

EDIT:

The following are the logs of oderer1 which returns this error:

$ kubectl logs ord1-hlf-ord-546fb7949b-zwl89 -n blockchain
2019-02-12 15:18:35.671 UTC [common/deliver] Handle -> DEBU 2fa Starting new deliver loop for 10.0.0.73:50228
2019-02-12 15:18:35.672 UTC [common/deliver] Handle -> DEBU 2fb Attempting to read seek info message from 10.0.0.73:50228
2019-02-12 15:18:35.674 UTC [orderer/common/server] Broadcast -> DEBU 2fc Starting new Broadcast handler
2019-02-12 15:18:35.674 UTC [orderer/common/broadcast] Handle -> DEBU 2fd Starting new broadcast loop for 10.0.0.73:50230
2019-02-12 15:18:35.674 UTC [orderer/common/broadcast] Handle -> WARN 2fe [channel: mychannel] Rejecting broadcast of message from 10.0.0.73:50230 with SERVICE_UNAVAILABLE: rejected by Consenter: will not enqueue, consenter for this channel hasn't started yet
2019-02-12 15:18:35.674 UTC [orderer/common/server] func1 -> DEBU 2ff Closing Broadcast stream
2019-02-12 15:18:35.675 UTC [grpc] Printf -> DEBU 300 transport: http2Server.HandleStreams failed to read frame: read tcp 10.0.0.10:7050->10.0.0.73:50230: read: connection reset by peer
2019-02-12 15:18:35.675 UTC [common/deliver] Handle -> WARN 301 Error reading from 10.0.0.73:50228: rpc error: code = Canceled desc = context canceled
2019-02-12 15:18:35.675 UTC [orderer/common/server] func1 -> DEBU 302 Closing Deliver stream
2019-02-12 15:20:59.484 UTC [orderer/common/server] Deliver -> DEBU 303 Starting new Deliver handler
2019-02-12 15:20:59.484 UTC [common/deliver] Handle -> DEBU 304 Starting new deliver loop for 10.0.0.73:52108
2019-02-12 15:20:59.484 UTC [common/deliver] Handle -> DEBU 305 Attempting to read seek info message from 10.0.0.73:52108
2019-02-12 15:20:59.487 UTC [orderer/common/server] Broadcast -> DEBU 306 Starting new Broadcast handler
2019-02-12 15:20:59.487 UTC [orderer/common/broadcast] Handle -> DEBU 307 Starting new broadcast loop for 10.0.0.73:52110
2019-02-12 15:20:59.487 UTC [orderer/common/broadcast] Handle -> WARN 308 [channel: mychannel] Rejecting broadcast of message from 10.0.0.73:52110 with SERVICE_UNAVAILABLE: rejected by Consenter: will not enqueue, consenter for this channel hasn't started yet
2019-02-12 15:20:59.487 UTC [orderer/common/server] func1 -> DEBU 309 Closing Broadcast stream
2019-02-12 15:20:59.488 UTC [grpc] Printf -> DEBU 30a transport: http2Server.HandleStreams failed to read frame: read tcp 10.0.0.10:7050->10.0.0.73:52110: read: connection reset by peer
2019-02-12 15:20:59.488 UTC [common/deliver] Handle -> WARN 30b Error reading from 10.0.0.73:52108: rpc error: code = Canceled desc = context canceled
2019-02-12 15:20:59.488 UTC [orderer/common/server] func1 -> DEBU 30c Closing Deliver stream

[Neekoy]

:)

Does it work if I scale down to 1 replica of Kafka though? I'm trying to get it running for test purposes at least, and set the Kafka replicas to 1. Now Kafka seems to be running properly, but I get the same error:

Error: got unexpected status: SERVICE_UNAVAILABLE -- will not enqueue, consenter for this channel hasn't started yet
command terminated with exit code 1

Does this mean it necessarily needs 4 Kafka replicas on different nodes, and will not run with one?

[Neekoy]

Not sure if it got created, and we couldn't really figure out the error. In the end we found the following repository and used a modified setup:

https://github.com/IBM/blockchain-network-on-kubernetes

We added kafka and zookeeper and root CA / intermediate CA to the above, and now it's working.

[alexvicegrab]

The IP address of the Ingress Controller (see its notes), is it the same as the CA's?

What are the logs of the Ingress Controller saying when you try access the CA with CURL?

[alexvicegrab]

And what do the CA logs say?

Finally, did you enroll the internal identity of the CA itself?

P.S. Thanks for the detailed info in the issue!

[Neekoy]

Thanks for the help @alexvicegrab. So the Ingress Controller has an AWS Load Balancer as the external-IP:

$ kubectl get services --namespace ingress-controller
NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP                                                                  PORT(S)                      AGE
nginx-ingress-controller        LoadBalancer   172.20.29.225   ae43432f2294d11e9a030021be5df150-1371082800.eu-central-1.elb.amazonaws.com   80:31866/TCP,443:31079/TCP   1d
nginx-ingress-default-backend   ClusterIP      172.20.205.17   <none>                                                                       80/TCP                       1d

When I point the domain orbix-fabric-ca-3.orbixpay.com there, I get the following error:

$ curl https://$CA_INGRESS/cainfo
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

The CA_POD definitely has the SSL certificate though:

$ kubectl exec -n blockchain $CA_POD -- cat /var/hyperledger/fabric-ca/msp/signcerts/cert.pem
-----BEGIN CERTIFICATE-----
MIICSzCCAfGgAwIBAgIUJKohVAXZxV/Uu/NSXfseWxnJfcEwCgYIKoZIzj0EAwIw
YzELMAkGA1UEBhMCSUUxDzANBgNVBAgTBkR1YmxpbjEOMAwGA1UEBwwFPG5pbD4x
ETAPBgNVBAoTCEFJRDpUZWNoMRMwEQYDVQQLEwpCbG9ja2NoYWluMQswCQYDVQQD
EwJjYTAeFw0xOTAyMDUxNDQ5MDBaFw0yMDAyMDUxNDU0MDBaMF0xCzAJBgNVBAYT
AlVTMRcwFQYDVQQIEw5Ob3J0aCBDYXJvbGluYTEUMBIGA1UEChMLSHlwZXJsZWRn
ZXIxDzANBgNVBAsTBmNsaWVudDEOMAwGA1UEAxMFYWRtaW4wWTATBgcqhkjOPQIB
BggqhkjOPQMBBwNCAAQhP8kh4RrL3+SKyvCJp8WGKC2N08bF/khe8lJT4AZZFAio
nrqayj9RQNB1CUmAffBoTYuC7Pk8PCVBouOqZcsvo4GIMIGFMA4GA1UdDwEB/wQE
AwIHgDAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBS62cPDLdgfHfDYYk5r34fDbqmY
3zAfBgNVHSMEGDAWgBQ7DHyE8P5hfuogIuhnblzu1AE6CDAlBgNVHREEHjAcghpj
YS1obGYtY2EtNzk4Yzk4Yzg0Zi1icmZscTAKBggqhkjOPQQDAgNIADBFAiEAjSXb
83lWKlDlDq2fGUUfZc4A0N4go97MSAlY4Tj1fOwCIDXwO14n66XFtn1MkhXzd79y
tlRlT3Y9vV1qv5OmZ1+X
-----END CERTIFICATE-----

So something's still not right :(

[alexvicegrab]

Aha, it looks like the issue is with the SSL then. The signcert is not the SSL certificate, but the ID certificate (created by the CA).

Does the CA secret exist, and can we check that certificate? If not, it might have to do with the Cert-manager

[alexvicegrab]

Let's get the logs of the Cert-manager also

[Neekoy]

Well I do have the CA secret, however it seems to just be a username/password pair, not a certificate:

$ kubectl get secrets -n blockchain
NAME                  TYPE                                  DATA   AGE
ca-hlf-ca--ca         Opaque                                2      1d
ca-postgresql         Opaque                                1      1d
default-token-zgkhv   kubernetes.io/service-account-token   3      1d

$ kubectl describe secret ca-hlf-ca--ca -n blockchain
Name:         ca-hlf-ca--ca
Namespace:    blockchain
Labels:       app=hlf-ca
              chart=hlf-ca-1.1.4
              heritage=Tiller
              release=ca
Annotations:  <none>

Type:  Opaque

Data
====
CA_ADMIN:     5 bytes
CA_PASSWORD:  24 bytes

And the logs for the Cert-Manager are the following:

$ kubectl logs cert-manager-7d4bfc44ff-jqx64 -n cert-manage
E0207 09:48:41.919899       1 reflector.go:205] github.com/jetstack/cert-manager/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Certificate: the server could not find the requested resource (get certificates.certmanager.k8s.io)
E0207 09:48:42.921472       1 reflector.go:205] github.com/jetstack/cert-manager/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Certificate: the server could not find the requested resource (get certificates.certmanager.k8s.io)
E0207 09:48:43.923059       1 reflector.go:205] github.com/jetstack/cert-manager/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Certificate: the server could not find the requested resource (get certificates.certmanager.k8s.io)

[alexvicegrab]

No, the TLS one: ca--tls

If you don't have it, the likely culprit is the Cert-manager

[alexvicegrab]

The logs support my conclusion as well

[Neekoy]

That would seem to be the case indeed. Thing is... how do we go about resolving this? The logs from the Cert-Manager don't really give me any ideas unfortunately.

[alexvicegrab]

Same here, but it is outside the scope of the webinar to debug the Cert-manager issue.

Perhaps Stack Overflow can help here, as it's likely other faced this issue.

One thing you may wish to try is to try Traefik instead of NGINX ingress. I have not used it myself but it is an ingress controller with integrated Certificate Management. You will need to update the annotations I'm guessing.

If the latter works, please ping back here so we can add it to the repo.

[Neekoy]

@alexvicegrab There's something I don't really understand -

I tried the FABRIC_CA_CLIENT_HOME part, and it says that the fabric-ca-client command is not found:

$ FABRIC_CA_CLIENT_HOME=./config fabric-ca-client getcacert -u https://$CA_INGRESS -M ./AidTechMSP
fabric-ca-client: command not found

I'm really not sure where to get this command from and it doesn't seem to be specified in the guide.

Or did you mean that it should be the content of the variable, like the following with quotes:

FABRIC_CA_CLIENT_HOME="./config fabric-ca-client getcacert -u https://$CA_INGRESS -M ./AidTechMSP"

[cagdast]

Hello @Neekoy. You should install Fabric CA Client, as well. Please see the 'install' section in this link.

[alexvicegrab]

Thanks @cagdast. @Neekoy, if you use a Mac you can also check our pyblic homebrew repos to easily install the binaries:
https://github.com/aidtechnology/homebrew-fabric-ca

[Neekoy]

Neat I did that and think I understand it. And how does one get this tool configtxgen - I can't seem to Google a setup for it.

[alexvicegrab]

Same thing, but https://github.com/aidtechnology/homebrew-fabric

The Hyperledger one is not being maintained, so you can:

brew tap aidtechnology/homebrew-fabric
brew install [email protected]

I've not gotten round to updating to 1.4.0, but will do in the next few weeks, unless someone submits a PR.

[Neekoy]

Awesome, and is there one for Ubuntu (apt package manager)?

[alexvicegrab]

Hmmm... I typically see this when I have some kind of mis-configuration of the network, but when I ran these commands (as I wrote them up for the Webinar), it all worked.

You may also wish to have a go at a similar but newer example in here:
https://github.com/aidtechnology/hgf-k8s-workshop

[alexvicegrab]

Quick Q, actually, how many nodes does your cluster have? It may be that Kafka has not started up correctly?

[Neekoy]

Omg your intuition is on point. xD

Kafka not getting started because of the affinity rules. Will try to comment them out real quick and try again.

[alexvicegrab]

Yes, or just increase the cluster size to at least 4 instances. In production you need at the very least 4 Kafka nodes, ideally more (for crash-tolerance).

I've seen a lot of these issues, so my intuition has developed accordingly, but still can't diagnose all issues as fast as I would like :)

[alexvicegrab]

Instead of that, you should just run a solo orderer without Kafka, see the Workshop for an example: https://github.com/aidtechnology/hgf-k8s-workshop

[Neekoy]

Well when I get to production it will be multiple nodes, I'm trying to get it going as a PoC. The good part is - Kafka is now up and running. The bad part is, I got another error that doesn't make much sense to me :(

kubectl exec -n blockchain $PEER_POD -- peer channel create -o ord1-hlf-ord.blockchain.svc.cluster.local:7050 -c mychannel -f /hl_config/channel/mychannel.tx

2019-02-12 16:10:35.246 UTC [msp] setupSigningIdentity -> DEBU 032 Signing identity expires at 2020-02-12 12:59:00 +0000 UTC
2019-02-12 16:10:35.246 UTC [msp] Validate -> DEBU 033 MSP AidTechMSP validating identity
2019-02-12 16:10:35.246 UTC [msp] GetDefaultSigningIdentity -> DEBU 034 Obtaining default signing identity
2019-02-12 16:10:35.246 UTC [grpc] Printf -> DEBU 035 parsed scheme: ""
2019-02-12 16:10:35.246 UTC [grpc] Printf -> DEBU 036 scheme "" not registered, fallback to default scheme
2019-02-12 16:10:35.246 UTC [grpc] Printf -> DEBU 037 ccResolverWrapper: sending new addresses to cc: [{ord1-hlf-ord.blockchain.svc.cluster.local:7050 0  <nil>}]
2019-02-12 16:10:35.246 UTC [grpc] Printf -> DEBU 038 ClientConn switching balancer to "pick_first"
2019-02-12 16:10:35.247 UTC [grpc] Printf -> DEBU 039 pickfirstBalancer: HandleSubConnStateChange: 0xc420220470, CONNECTING
2019-02-12 16:10:35.258 UTC [grpc] Printf -> DEBU 03a pickfirstBalancer: HandleSubConnStateChange: 0xc420220470, READY
2019-02-12 16:10:35.258 UTC [channelCmd] InitCmdFactory -> INFO 03b Endorser and orderer connections initialized
2019-02-12 16:10:35.258 UTC [msp] GetDefaultSigningIdentity -> DEBU 03c Obtaining default signing identity
2019-02-12 16:10:35.259 UTC [msp] GetDefaultSigningIdentity -> DEBU 03d Obtaining default signing identity
2019-02-12 16:10:35.259 UTC [msp/identity] Sign -> DEBU 03e Sign: plaintext: 0ACF070A0A416964546563684D535012...120E0A0C4D79436F6E736F727469756D 
2019-02-12 16:10:35.259 UTC [msp/identity] Sign -> DEBU 03f Sign: digest: B759F3E7EBCF0FD0DEA2F759AB86F1A5389283C34F1A813488C7D115657FB807 
2019-02-12 16:10:35.259 UTC [msp] GetDefaultSigningIdentity -> DEBU 040 Obtaining default signing identity
2019-02-12 16:10:35.259 UTC [msp] GetDefaultSigningIdentity -> DEBU 041 Obtaining default signing identity
2019-02-12 16:10:35.259 UTC [msp/identity] Sign -> DEBU 042 Sign: plaintext: 0A86080A1508021A0608FBDF8BE30522...4ED3E18BFD2A512268C358908CDBB965 
2019-02-12 16:10:35.259 UTC [msp/identity] Sign -> DEBU 043 Sign: digest: 99A41CA3C4D7E60FB848C01422A6441A9CA04B23C0837A0A0F4BFF1D3306C2CA 
2019-02-12 16:10:35.259 UTC [grpc] Printf -> DEBU 044 parsed scheme: ""
2019-02-12 16:10:35.259 UTC [grpc] Printf -> DEBU 045 scheme "" not registered, fallback to default scheme
2019-02-12 16:10:35.259 UTC [grpc] Printf -> DEBU 046 ccResolverWrapper: sending new addresses to cc: [{ord1-hlf-ord.blockchain.svc.cluster.local:7050 0  <nil>}]
2019-02-12 16:10:35.259 UTC [grpc] Printf -> DEBU 047 ClientConn switching balancer to "pick_first"
2019-02-12 16:10:35.259 UTC [grpc] Printf -> DEBU 048 pickfirstBalancer: HandleSubConnStateChange: 0xc420254810, CONNECTING
2019-02-12 16:10:35.305 UTC [grpc] Printf -> DEBU 049 pickfirstBalancer: HandleSubConnStateChange: 0xc420254810, READY
Error: got unexpected status: BAD_REQUEST -- error authorizing update: error validating ReadSet: readset expected key [Group]  /Channel/Application at version 0, but got version 1
command terminated with exit code 1

This is with all the services being up and running:

$ kubectl get pods -n blockchain
NAME                                     READY   STATUS    RESTARTS   AGE
ca-hlf-ca-7694b568d4-tq6vs               1/1     Running   0          7h
ca-postgresql-0                          1/1     Running   0          7h
cdb-peer1-hlf-couchdb-5b5bf7d89d-hg774   1/1     Running   0          3m
kafka-hlf-0                              2/2     Running   1          14m
kafka-hlf-1                              2/2     Running   0          12m
kafka-hlf-2                              2/2     Running   0          12m
kafka-hlf-3                              2/2     Running   0          11m
kafka-hlf-exporter-6c7c5cc787-v9hxt      1/1     Running   3          14m
kafka-hlf-zookeeper-0                    1/1     Running   0          14m
ord1-hlf-ord-546fb7949b-zwl89            1/1     Running   0          1h
peer1-hlf-peer-56f69f6964-wdjvp          1/1     Running   0          2m

[alexvicegrab]

Not sure what the issue is. But initial misconfiguration of the Kafka may be the culprit.

I'm developing a library for automatically deploying networks, but it's in alpha:
https://github.com/aidtechnology/nephos

[alexvicegrab]

Rereading this once more:

error validating ReadSet: readset expected key [Group]  /Channel/Application at version 0, but got version 1

Is it possible that the channel has already been created?

In either case, were you able to resolve this?