Deploy Duration about viya4-monitoring-kubernetes

[BryanEllington]

1st deployment has cluster-logging and cluster-monitoring - This gets you cluster-wide metrics and logs
nth deployment has viya-monitoring, but no cluster-logging and no cluster-monitoring - This deploys viya and enables that viya namespace for monitoring

[BryanEllington]

Monitoring should only take 2-3 minutes. Logging has a lot of conservative sleeps waiting for components to start. We could improve on that by polling and checking actual status instead.

The fundamental difference is that Viya is quick to deploy since it just creates the Kubernetes resources. It can take 20-40 minutes for everything to actually start and be ready for logins. For monitoring and logging, the components are up and running when the scripts complete, so we're not comparing apples to apples here.

[andybouts]

I don't disagree that it's apples :: oranges, and maybe monitoring only does take a few minutes. But it's pretty frustrating for an end user trying to add Viya configurations (transformers, etc) into ../site-config/ and test it only to wait +/-15 mins to see a failure and then try to resolve it (sometimes even trying to remembering what you did 15 mins ago) because the logging deployment step takes so long.

[BryanEllington]

I would recommend installing cluster monitoring and logging first, though. That gives you the ability to monitor the Viya deployment. Enabling monitoring of the Viya namespace then takes < 30 sec.

That said, I definitely agree that the deployment times are high and it's something we need to improve.

[andybouts]

Are you saying I could do it 2 steps ...
Step 1, initially deploy logging and monitoring
Step 2, deploy viya (logging and monitoring will not be un-deployed?)
Step n, re-deploy Viya (logging and monitoring will not be un-deployed?)

[BryanEllington]

I'm not sure exactly what directions/projects you're using to deploy, so I don't want to mislead you, but basically yes:

1. Deploy monitoring and logging (monitoring/bin/deploy_monitoring.sh, logging/bin/deploy_logging_open.sh)
2. Deploy Viya
3. Enable the new Viya deployment for monitoring (monitoring/bin/deploy_monitoring_viya.sh)

[andybouts]

I am using the GitHub deployment project with Docker deploy command ...

So for step 1, I would have --tags "baseline,cluster-logging,cluster-monitoring,install"
Then for step 2, I would have --tags "baseline,viya,install" ... it should run in a few mins instead of 15+/- mins
Then for step n, I would re-do this step n number of times and it should run in a few mins instead of 15+/- mins

docker run --rm \
  --group-add root \
  --user $(id -u):$(id -g) \
  --volume $HOME/viya4-deployment:/data \
 <truncated>
  viya4-deployment --tags "baseline,viya,cluster-logging,cluster-monitoring,install" > viya.deploy.log

[BryanEllington]

Modify step 2 to include viya-monitoring:
--tags "baseline,viya,viya-monitoring,install"
That enables the newly created Viya namespace for monitoring. The viya-monitoring deployment should only take ~30 sec.

[andybouts]

oh, so just remove the cluster-logging for the 2nd and Nth deployment?

[andybouts]

hey @BryanEllington , it looks like the suggestion worked ...

Today in my green env redeploying for the Nth time took 13+ mins:

command: 
  viya4-deployment --tags "baseline,viya,cluster-logging,cluster-monitoring,install" > viya.deploy.log

results:
PLAY RECAP *********************************************************************
localhost                  : ok=119  changed=33   unreachable=0    failed=0    skipped=66   rescued=0    ignored=0

Monday 22 November 2021  14:36:50 +0000 (0:00:00.221)       0:13:19.984 *******
===============================================================================
monitoring : cluster-logging - deploy --------------------------------- 682.88s
vdm : kustomize - Generate deployment manifest ------------------------- 35.17s
vdm : manifest - deploy ------------------------------------------------ 13.27s

[andybouts]

then, today in my gray env, deploying for the Nth time -- it took just 2+ mins:

command:
  viya4-deployment --tags "baseline,viya,viya-monitoring,install" > viya.deploy.log

results:
PLAY RECAP *********************************************************************
localhost                  : ok=111  changed=31   unreachable=0    failed=0    skipped=65   rescued=0    ignored=0

Monday 22 November 2021  14:41:14 +0000 (0:00:00.239)       0:02:03.614 *******
===============================================================================
vdm : kustomize - Generate deployment manifest ------------------------- 37.99s
vdm : manifest - deploy ------------------------------------------------ 14.09s
vdm : manifest - deploy update ----------------------------------------- 10.88s

[gsmith-sas]

As shown in the output @andybouts shared, omitting the cluster level tasks of cluster-logging (which deploys the cluster-level log monitoring components) and cluster-monitoring (which deploys the cluster-level metrics monitoring components) reduced deployment times by 84%. As @BryanEllington explained, those tasks handle things that only need to be done once per cluster. Closing this issue.