kyma-project / compass-manager Goto Github PK

Description

For our release management and to fulfil SAP product standards, we have to document how our testing strategy for the Compass Manager looks like.

Some example links to such documentations are available here: https://wiki.one.int.sap/wiki/display/kyma/Testing+Strategy+-+Link+summary

For the AC, the testing strategy is already documented.

AC:

• Create a testing strategy document for the Compass Manager (proposal for the location: https://github.com/kyma-project/infrastructure-manager/tree/main/docs/contributor/...-ci-cd.md )
• When the documentation is ready, update the link in this Wiki page: https://wiki.one.int.sap/wiki/display/kyma/Testing+Strategy+-+Link+summary

Area
Compass Manager

Reasons

Mandatory part of the delivery process and required for a fast creation of Microdeliveries.

Assignees

@kyma-project/technical-writers

Attachments

Description

Configure a markdown link checker that will ensure that links we use in our *.MD files are valid.

Reasons

• Documentation consistency
• Good developer experience

Attachments

• kyma-project/community#834
• kyma-project/infrastructure-manager#34

/area documentation
/area application-connector
/kind feature

We need to design and perform manual stress test that will help us to learn what is the scalability of the Compass Manager.

Acceptance criteria:

• Prepare a short document describing how test will be performed
• Execute the experiment with various number of input data (Kyma clusters)
• Document the results

Note: we can mock Director calls to not provide stress on Compass system. We would need a service with three endpoints to do so: register, unregister, and get token.

Deprecated: Use WebhookServer instead. A WebhookServer can be created via webhook.NewServer

deprecated docs
new docs

https://github.com/kyma-project/compass-manager/blob/10d6039aa73fbef6807d9dab91a37b47bfdc5ff0/main.go#L90C5-L90C5

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-reuse_tool-2
Explanation: Does it have LICENSES directory with licenses? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-reuse_tool-4
Explanation: Is it compliant with REUSE rules? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

Description

To avoid spoofing attacks, any communication between the Compass Manager and 3rd party services has to be encrypted.. The Compass Manager has to enforce and strictly validate the TLS communication to 3rd party services (e.g. Compass Director GraphQL and access to Kubernetes API on KCP and SKR side).

AC:

• Verify the communication to the Compass Director GraphQL service and enforce + validate the TLS communication used by the client API
• Verify the communication to the KCP / SKR Kubernetes API and make sure an encrypted and validated communication is used by the Kubernetes Client API.

Steps to exploit

Attacker can use a man-in-the-middle attack to spoof network traffic.

Risk assessment

Part of the Threat Modelling workshop from 2023-11-29.

Proposed mitigation

Review the used source code and ensure that only encrypted communication to the 3rd party service (e.g. Kubernetes API, Compass Directory GraphQL) is possible and enforce a strict validation of the SSL certificate.

Description

An attacker could inject malicious cause into consumed 3rd party services and Compass Manager is receiving risky payloads from these systems.

We have to ensure that payload of 3rd party services is, even if it's malicious, won't lead to security problems for the. Compass Manager.

AC:

• The Compass Manager has to be able to deal with unexpected huge payload properly by adding upload-limits (e.g. max. 2MB per HTTP response)
• The payload of responses has to be evaluated and discarded if it's not compliant with the agreed technical contract (e.g. apply data validation via JSONSchema, if JSON is expected discard any other input format etc.)
• If data of received payload is used for further processing, a sanity check has to happen for these data (e.g. max-string lengh, check for patterns etc.) and they have to be treated as untrusted data.

Steps to exploit

Attacker injects unexpected payload into the response of a 3rd party system.

Risk assessment
Part of the Threat Modelling workshop from 2023-11-29.

Proposed mitigation

Establish security mechanisms to handle malicious properly.

Description

As pre-requisite for the rollout of the Compass Manager to KCP STAGE and PROD is the preparation and execution of this migration step mandatory.

Migrate old runtime which are already registered at the Compass Director and create compass mapping for it.

AC:

• Implement migration logic
• Inform @zhoujing2022 about the date when the migration for
  • KCP STAGE will be completed
  • KCP PROD will be completed

Reasons

We need to make a checkpoint after a major implementation and check on the target environment if all assumptions have been met and that they are operational.

Acceptance criteria

• Compass Manager must have the option to register and unregister runtime from Compass disabled. That logic should be mocked and only return logs/printf,
• Compass Mapping CR should be created during tests,
• the core functionality and flow must remain untouched,
• environment must be cleaned after tests

Test scenarios

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-vulnerability_alerts-1
Explanation: Are vulnerability alerts enabled? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

Description

Log output of the Compass Manager could include sensitive data which enables attackers to infiltrate the application. A code review has to be applied to ensure that no sensitive application data (e.g. tokens, passwords, personal data etc.) will be logged by our software.

AC:

• Review the log-output of the Compass Manager and ensure that no sensitive data will be disclosed.

Steps to exploit

Attacker reviews the log output, finds sensitive data and abuses it against us.

Risk assessment
Part of the Threat Modelling workshop from 2023-11-29.

Proposed mitigation

Review any log output of the application to ensure that no sensitive data can be disclosed as part of log messages.

Description

To be ready for the go-live, we have to create an on-call guide for the Compass Manager. This is also a pre-requisite for the Microdelivery (#4901) of the Compass Manager.

Possible location for the on-call guide: https://github.tools.sap/kyma/documentation/tree/main/kyma-internal/on-call-guides/mps

AC:

• Document the common use-cases / possible incidents we have to expect when the Compass Manager runs in a productive context

Area

• Compass Manager

Reasons

Mandatory pre-requisite before we can go-live and part of the SAP Product Standards.

Assignees

@kyma-project/technical-writers

Attachments

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-reuse_tool-1
Explanation: Does README mention REUSE? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

Description

To ensure a controlled rollout of the Compass Manager, this cut-over plan is used to define the required steps, their order and owner:

• Adding Metrics support to Compass Manager, Owner: @kyma-project/framefrog
• Dry-Run on STAGE completed and results reviewed, Owner: SRE / @kyma-project/framefrog
• Enabling Compass Manager on STAGE,Owner: SRE
• Review with SRE required steps for PROD rollout, Owner: SRE / @kyma-project/framefrog

Requires

#87

Description

As Prow will be discontinued in 2024, we have to move the Prow jobs used for the provisioner to an alternative CI/CD system. In our case Github Actions is the preferred choice.

Overview of all existing Prow-jobs is listed here: https://github.com/search?q=repo%3Akyma-project%2Ftest-infra+framefrog&type=code&p=1

AC:

• Identify which of the jobs listed in the URL above are required during the Compass Manager development-lifecycle and relevant in the longterm (have to be migrated)
• All Compass Manager related Prow jobs are migrated to Github Actions

Reasons

Migrate CI/CD jobs from Prow to Github Actions as Prow will be discontinued in 2024.

Attachments

Description

To avoid that malicious code can be injected into our productive systems by injecting untrusted container images, we have to verify our build and delivery process and ensure that

• all container images are build following SAP SLC-29 compliancy guidelines (using trusted build infrastructure, sign images)
• the publishing of container images happens only via trusted container registries
• consumption/deployments of container images are using only trusted container registries

AC:

• Ensure that our build pipeline is using an SLC-29 compliant build-mechanism to product container images (image is also signed by the SAP Signify service)
• Verify that any produced container image is only published and consumed from a trusted container registry (review of deployment manifests)

Steps to exploit

Attacker find a way to inject malicious code into our productive systems because container images were build on untrusted systems, not signed and published to untrusted container registries.

Risk assessment
Part of the Threat Modelling workshop from 2023-11-29.

Proposed mitigation

Review the build process of our container images and ensure they are build by using SLC-29 compliant build-pipelines and getting signed by SAP Signify. Verify also the deployment manifests and ensure that only images from trusted container registries will be deployed.

Implement code that will register Runtime in the director.

Acceptance criteria:

• Compass Manager is able to make authorised calls to the Compass Director API
• Compass Manager registers new Runtime when Kyma CR contains a new kubeconfig
• It must be assumed that Compass may be temporarily unavailable. We must provide resilience mechanisms, and eventual consistency.
• Compass Manager unregisters Runtime when Kyma CR is removed.

Description

Compass Manager should provide metrics to allow early issues detection.

Reasons

Compass Manager is a component that is responsible for Compass integration. In case of a downtime the impact on Kyma Control Plane will be significant. We must prevent that by increasing the observability.

Implement reconciliation loop that watches Kyma custom resource.

Acceptance criteria:

• Reconciliation is triggered only if one of the the following events occurs:
  • New Kyma CR created containing kubeconfig (Bring Your Own cluster scenario)
  • Kubeconfig field set on the existing Kyma CR
• Reconciliation is not triggered if:
  • The last reconciliation attempt for particular Runtime succeeded (the Runtime is already registered in the Compass)
• Functionality of registering Runtime in Compass, and creating Config Map for Compass Runtime Agent is mocked
  - [ ] If Kyma resource was removed, the Runtime must be unregistered.

Description

Currently, the Compass Manager uses the context.TODO or context.Background instead of the context from the Reconcile loop. We need to change the logic to properly pass to the context function and use it.

Description

To follow the least privilege approach for the Compass Manager, restrictive RBAC rules have to be established.

The Kubernetes service account which is used to run the Compass Manager should only be allowed to access Kubernetes resources which are mandatory to fulfil his job. Access to other resources on the KCP cluster has to be rejected.

AC:

• Define RBAC rules (role + role bindings) for the service account used by the Compass Manager
  • Per default is any access to KCP resources blocked
  • Whitelist the access only for Kubernetes resources which are required by the Compass Manager to work correctly.

Steps to exploit

Compass Manager is able to read Kubernetes resources which are NOT required to fulfil his work.

Risk assessment

Result of the Treat Modelling workshop from 2023-11-29.

Proposed mitigation

Define RBAC rules for the Compass Manager service account by setting up a proper and restrictive Kubernetes role and role binding.

Description

Compass Manager should not create separate Kubernetes clients for each SKR, but use a solution to be provided by the lifecycle-manager. That solution will enable central synchronisation and k8s resource creation on client clusters. Information on progress as well as architectural decisions can be found in this ADR.

Additional information

For the first release of Compass Manager, we will not use this approach because it has not yet been implemented. Eventually, we will dispose of our own Kubernetes client for SKR, and replace it with the central approach proposed in ADR (mentioned above)

Description

With #28 we are able to make the compass manager transparent and also simplify our operational life by establishing smart metrics and alerting rules.

Goals of this task is to identify which metrics / KPIs are business relevant and what the critical threshold for it are. We also have to define an action plan when such a threshold is reached which trigger a required action to bring our business back on track. Finally, alerting rules have to be configured which inform us as soon as one of the thresholds is reached.

AC:

• Think about technical and business critical metrics / KPIs which give a clear indication of the quality and health of the Compass Manager
  • Define the reason why this metric is relevant and what it represents.
  • Define the threshold (min <> max etc.) which indicate an service degradation or health issue of the Compass Manager. If a metric has no threshold, verify if it's for us still helpful to measure this value.
  • Specify the required action that has to be applied if a threshold is reached to recover the Compass Manager into a productive and healthy state
  • Present the results in the team to collect the feedback of the colleagues.
• Implement the identify business metrics in the Compass Manager
• Configure alerting rules which inform the team as soon as one of the thresholds is reached

Reasons

Improve operational quality and simplify on-call shifts by establish proper metrics/KPI measuring and alerting.

Attachments

Description
Prepare a Go program/script that will iterate over Kyma resources. For each Kyma resource it will:

1. Read runtime-id label from the Kyma resource
2. Set compass-runtime-id-for-migration annotation with value for runtime-id label

Reasons

In order to migrate to the architecture with the Compass Manager responsible Compass Integration some additional steps must be performed. Before Compass Manager will be deployed on the target environment there will be a need to modify existent Kyma resources. The migration script is needed to make sure Compass Manager will not attempt to register runtimes that are already registered.

In the current architecture the Provisioner is responsible for registering Runtimes and preparing Config Map for Compass Runtime Agent. There is a need to prepare a detailed plan how to switch from the current architecture to the new one based on the Compass Manager.

Cover the following:

• What will be the phases of migration?
• Would there be a need to distinguish Kyma resources representing new and migrated Runtimes? Would KEB need to behave differently?
• Consider whether we need to perform any steps in a maintenance window?

Description

CMP system used for registering applications on KCP DEV will be replaced by a new system until 12th Jan 2024 which has a new URL.

AC:

• Verify the secret generated by the Provisioner (used by the Application Connector) and ensure that the value of existing secrets gets updated (if an update is required)
  • Implement the migration logic for the application connector to work after the CMP URL update
• Update the Compass Manager on DEV to use the new CMP system URL: compass-gateway-auth-oauth.cmp-main.dev.kyma.cloud.sap
• Ping Nikolay Y. to create IntegrationSystem/s and to issue credentials for us (see SlackURL in the issue field "external link")

Expected result

KCP DEV is using the new CMP system URL.

Actual result

We use the old CMP system URL on KCP DEV which will be replaced until 12th Jan 2024.

Steps to reproduce

Verify configuration of Compass client on KCP DEV.

Troubleshooting

Description

To get rid of manual testing efforts and to ensure a consistent high quality of our deliverables, an automated end-2-end test for the Compass Manager is required.

Scope of the test is to verify that the Compass Manager could be successfully installed in a local Kubernetes cluster (e.g. k3d) and is able to interact successfully with a Compass-Director stub and the remote SKR cluster.

AC:

• Deploy the Compass Manager and a stub of the Compass Director in a local Kubernetes cluster (e.g. K3d cluster)
• Simulate the activation of the application connector module on an SKR (e.g. it's fine to use the same local K3d cluster to simulate an SKR instance)
• Verify that the Compass Directory is reacting on the App-Conn module activation and interacts with the Compass Director stub
• Simulate a successful application registration (happy path)
• Simulate a failing application registration and check that the retry mechanism of the Compass Manager works as expected (unhappy path)
• Simulate the unregistration of an application (TBC if this scenario is worth to be tested)

Reasons

Reduce manual testing efforts and ensure consistent high quality deliveries of the Compass Manager.

Attachments

We must consider creating component test for Compass Manager. Such a test could work in a similar way as Compass Runtime Agent test. It could register itself in DEV Compass.

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-vulnerability_alerts-1
Explanation: Are vulnerability alerts enabled? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

Description

Create developer-facing documentation

Reason
We need to have documentation that describes how to deploy, test, and utilize Compass Manger

Implement code that will create Secret for configuring Compass Runtime Agent.

Acceptance criteria:

• kyma-project/lifecycle-manager#991 Proposal with unifying the way of inserting k8s objects from central components to customer SKR.
  After team disscusion -> For the first release of Compass Manager, we will not use this approach because it has not yet been implemented. Eventually, we will dispose our own Kubernetes client for SKR, and replace it with the central approach proposed in this issue.
• When Runtime was successfully registered Compass Manager performs the following actions:
  • creates connection token that will be used by the Compass Runtime Agent.
  • creates Secret containing the following fields:
    • CONNECTOR_URL
    • TOKEN
    • RUNTIME_ID
    • TENANT
• It must be assumed that Compass may be temporarily unavailable. We must provide resilience mechanisms, and eventual consistency.