When deploying vespa ha on self-built kubernetes, it reports that the host name cannot be resolved. about sample-apps

[kkraune]

Hi, does the sample app work if you run it with no changes?

[TonyAnn]

Hi, does the sample app work if you run it with no changes?

helllo kkraune I think the current problem is caused by the inability to resolve the dns in the container; I only modified the service, which did not have much impact on the container itself.

[TonyAnn]

hi kkraune How to deploy vespa in self-built kubernetes. Is there any better reference document or helm chart available? Instead of gke’s document link.

[kkraune]

Hi, my question was, can you make the guide work without changing anything in the source files? It is hard to help otherwise.

You can also try https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/basic-search-on-gke

I am sorry we do not have better examples and documentation on this, submissions welcome :-)

[bratseth]

Vespa requires the network incl DNS to work before nodes are started so the usual issue is that people try to both start the containers and Vespa at the same time.

Lots of people are running Vespa on k8s at large scale so its resolvable, but we can't really provide detailed help with it.

In general you'll be better off using Vespa Cloud as the control plane rather than a generic control plane like k8s - you can still deploy it in your own AWS account/GCP project.

[TonyAnn]

Hi, my question was, can you make the guide work without changing anything in the source files? It is hard to help otherwise.

You can also try https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/basic-search-on-gke

I am sorry we do not have better examples and documentation on this, submissions welcome :-)

hello kkraune
Without modifying any configuration, after installing vespa, the deployment application reports the following warning log

[root@obtest133 gke]# zip -r - . -x README.md "config/*" | curl --header Content-Type:application/zip --data-binary @- http://localhost:19071/application/v2/tenant/default/prepareandactivate
adding: hosts.xml (deflated 83%)
adding: schemas/ (stored 0%)
adding: schemas/music.sd (deflated 57%)
adding: services.xml (deflated 71%)
{
"log": [{
"time": 1710465277829,
"level": "WARNING",
"message": "Host named 'vespa-feed-container-0.vespa-internal.default.svc.cluster.local' may not receive any config since it differs from its canonical hostname '100-65-143-29.vespa-feed.default.svc.cluster.local' (check DNS and /etc/hosts).",
"applicationPackage": true
}, {
"time": 1710465277831,
"level": "WARNING",
"message": "Host named 'vespa-feed-container-1.vespa-internal.default.svc.cluster.local' may not receive any config since it differs from its canonical hostname '100-94-197-63.vespa-feed.default.svc.cluster.local' (check DNS and /etc/hosts).",
"applicationPackage": true
}, {
"time": 1710465277832,
"level": "WARNING",
"message": "Host named 'vespa-query-container-0.vespa-internal.default.svc.cluster.local' may not receive any config since it differs from its canonical hostname '100-65-143-15.vespa-query.default.svc.cluster.local' (check DNS and /etc/hosts).",
"applicationPackage": true
}, {
"time": 1710465277833,
"level": "WARNING",
"message": "Host named 'vespa-query-container-1.vespa-internal.default.svc.cluster.local' may not receive any config since it differs from its canonical hostname '100-94-197-20.vespa-query.default.svc.cluster.local' (check DNS and /etc/hosts).",
"applicationPackage": true
}],
"message": "Session 16 for tenant 'default' prepared and activated.",
"session-id": "16",
"activated": true,
"tenant": "default",
"url": "http://localhost:19071/application/v2/tenant/default/application/default/environment/prod/region/default/instance/default",
"configChangeActions": {
"restart": [],
"refeed": [],
"reindex": []
}
}

[kkraune]

I just ran all the steps in https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA/gke, and it worked fine for me

Make sure all config servers are started - this takes 2-3 minutes:

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
vespa-configserver-0   1/1     Running   0          12m
vespa-configserver-1   1/1     Running   0          11m
vespa-configserver-2   1/1     Running   0          10m

Then

$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
vespa-admin-0             1/1     Running   0          2m29s
vespa-configserver-0      1/1     Running   0          15m
vespa-configserver-1      1/1     Running   0          14m
vespa-configserver-2      1/1     Running   0          13m
vespa-content-0           1/1     Running   0          2m28s
vespa-content-1           1/1     Running   0          2m12s
vespa-feed-container-0    1/1     Running   0          2m28s
vespa-feed-container-1    1/1     Running   0          2m27s
vespa-query-container-0   1/1     Running   0          2m28s
vespa-query-container-1   1/1     Running   0          2m26s

and then go through all the steps in the readme to run the healthchecks - it takes some time to start the Vespa services inside the pods

$ curl http://localhost:19107/state/v1/health
{"status":{"code":"down","message":"matchengine: Search interface is offline"}}

$ curl http://localhost:19107/state/v1/health
{"status":{"code":"down","message":"matchengine: Search interface is offline"}}

$ curl http://localhost:19107/state/v1/health
{"status":{"code":"up"}}

So this works at least in the Google Cloud environment. Please see https://docs.vespa.ai/en/operations-selfhosted/node-setup.html#hostname and linked articles from there for the hostname problems you might have. Hope this helps

[TonyAnn]

hi kkraune ， In my cluster, the following detections are normal. Can this prove that the HA deployment is ready now?
1.pod component
[root@obtest133 ~]# kubectl get pod |grep -i vespa
vespa-admin-0 1/1 Running 0 3d1h
vespa-configserver-0 1/1 Running 0 3d2h
vespa-configserver-1 1/1 Running 0 3d2h
vespa-configserver-2 1/1 Running 0 3d2h
vespa-content-0 1/1 Running 0 3d1h
vespa-content-1 1/1 Running 0 3d1h
vespa-feed-container-0 1/1 Running 0 3d1h
vespa-feed-container-1 1/1 Running 0 3d1h
vespa-query-container-0 1/1 Running 0 3d1h
vespa-query-container-1 1/1 Running 0 3d1h

2. Cluster status check
   kubectl port-forward pod/vespa-configserver-0 19071
   [root@obtest133 ~]# curl http://localhost:19071/state/v1/health
   {
   "time" : 1710731499447,
   "status" : {
   "code" : "up"
   },
   "metrics" : {
   "snapshot" : {
   "from" : 1.710731438526E9,
   "to" : 1.710731498526E9
   },
   "values" : [ {
   "name" : "requestsPerSecond",
   "values" : {
   "count" : 0,
   "rate" : 0.0
   }
   }, {
   "name" : "latencySeconds",
   "values" : {
   "average" : 0.006,
   "sum" : 0.0,
   "count" : 0,
   "last" : 0.006,
   "max" : 0.006,
   "min" : 0.006,
   "rate" : 0.0
   }
   } ]
   }

[TonyAnn]

Vespa requires the network incl DNS to work before nodes are started so the usual issue is that people try to both start the containers and Vespa at the same time.

Lots of people are running Vespa on k8s at large scale so its resolvable, but we can't really provide detailed help with it.

In general you'll be better off using Vespa Cloud as the control plane rather than a generic control plane like k8s - you can still deploy it in your own AWS account/GCP project.

hello bratseth，Due to company security restrictions, hosted public cloud services cannot be used and self-built clusters are required.

[bratseth]

Due to company security restrictions, hosted public cloud services cannot be used and self-built clusters are required.

Not when the data plane remains in an account owned by your company though?