mspnp / aks-baseline

This is the Azure Kubernetes Service (AKS) Baseline Cluster reference implementation as produced by the Microsoft Azure Architecture Center.

Azure Kubernetes Service (AKS) baseline cluster

This reference implementation demonstrates the recommended starting (baseline) infrastructure architecture for a general purpose AKS cluster. This implementation and document is meant to guide an interdisciplinary team or multiple distinct teams like networking, security and development through the process of getting this general purpose baseline infrastructure deployed and understanding the components of it.

We walk through the deployment here in a rather verbose method to help you understand each component of this cluster, ideally teaching you about each layer and providing you with the knowledge necessary to apply it to your workload.

Azure Architecture Center guidance

This project has a companion set of articles that describe challenges, design patterns, and best practices for a secure AKS cluster. You can find this article on the Azure Architecture Center at Azure Kubernetes Service (AKS) baseline cluster. If you haven't reviewed it, we suggest you read it as it will give added context to the considerations applied in this implementation. Ultimately, this is the direct implementation of that specific architectural guidance.

Architecture

This architecture is infrastructure focused, more so than on workload. It concentrates on the AKS cluster itself, including concerns with identity, post-deployment configuration, secret management, and network topologies.

The implementation presented here is the minimum recommended baseline for most AKS clusters. This implementation integrates with Azure services that deliver observability, provide a network topology that support multiregional growth, and keep the in-cluster traffic secure as well. This architecture should be considered your starting point for pre-production and production stages.

The material here is relatively dense. We strongly encourage you to dedicate time to walk through these instructions, with a mind to learning. Therefore, we do NOT provide any "one click" deployment here. To understand the relationship between the deployed resources, we suggest that you consult the detailed architecture overview while exploring your deployment. Once you've understood the components involved and identified the shared responsibilities between your team and your great organization, it is encouraged that you build suitable, auditable deployment processes around your final infrastructure.

Throughout the reference implementation, you will see reference to Contoso Bicycle. It is a fictional small and fast-growing startup that provides online web services to its clientele on the west coast of North America. They have no on-premises datacenters and all their containerized line of business applications are now about to be orchestrated by secure, enterprise-ready AKS clusters. You can read more about their requirements and their IT team composition. This narrative provides grounding for some implementation details, naming conventions, and so on. You should adapt as you see fit.

Finally, this implementation uses the ASP.NET Core Docker sample web app as an example workload. This workload is purposefully uninteresting, as it is here exclusively to help you experience the baseline infrastructure.

Core architecture components

Azure platform

• AKS v1.30
  • System and user node pool separation
  • AKS-managed Microsoft Entra ID integration
  • Microsoft Entra ID-backed Kubernetes RBAC (local user accounts disabled)
  • Managed identities
  • Azure CNI
  • Azure Monitor for containers
• Azure virtual networks (hub-spoke)
  • Azure Firewall managed egress
• Azure Application Gateway (WAF)
• AKS-managed internal load balancers

In-cluster OSS components

• Azure Workload Identity [AKS-managed add-on]
• Flux GitOps Operator [AKS-managed extension]
• ImageCleaner (Eraser) [AKS-managed add-on]
• Kubernetes Reboot Daemon
• Secrets Store CSI Driver for Kubernetes [AKS-managed add-on]
• Traefik Ingress Controller

Also do not forget to view the detailed architecture diagram to understand how the deployed resources work together in this reference architecture.

Deploy the reference implementation

A deployment of AKS-hosted workloads typically involves a separation of duties and lifecycle management in the areas of prerequisites, the host network, the cluster infrastructure, and finally the workload itself. Different teams often are responsible for each of these components. This reference implementation follows a similar approach. Also, be aware our primary purpose is to illustrate the topology and decisions of a baseline cluster. We feel a "step-by-step" flow will help you learn the pieces of the solution and give you insight into the relationship between them. Ultimately, lifecycle/SDLC management of your cluster and its dependencies will depend on your situation (team roles, organizational standards, and so on), and will be implemented as appropriate for your needs.

Please start this learning journey in the Preparing for the cluster section. If you follow this through to the end, you'll have our recommended baseline cluster installed, with an end-to-end sample workload running for you to reference in your own Azure subscription.

1. 🚀 Prepare for the cluster

There are considerations that must be addressed before you start deploying your cluster. Do I have enough permissions in my subscription and AD tenant to do a deployment of this size? How much of this will be handled by my team directly vs having another team be responsible?

• Begin by ensuring you install and meet the prerequisites
• Procure client-facing and AKS Ingress Controller TLS certificates
• Plan your Microsoft Entra ID integration

2. 🔌 Build target network

Microsoft recommends AKS be deployed into a carefully planned network; sized appropriately for your needs and with proper network observability. Organizations typically favor a traditional hub-spoke model, which is reflected in this implementation. While this is a standard hub-spoke model, there are fundamental sizing and portioning considerations included that should be understood.

• Build the hub-spoke network

3. 📦 Deploy the cluster

This is the heart of the guidance in this reference implementation; paired with prior network topology guidance. Here you will deploy the Azure resources for your cluster and the adjacent services such as Azure Application Gateway WAF, Azure Monitor, Azure Container Registry, and Azure Key Vault. This is also where you will validate the cluster is bootstrapped.

• Prep for cluster bootstrapping
• Deploy the AKS cluster and supporting services
• Validate cluster bootstrapping

We perform the prior steps manually here for you to understand the involved components, but we advocate for an automated DevOps process. Therefore, incorporate the prior steps into your CI/CD pipeline, as you would any infrastructure as code (IaC). See the dedicated AKS baseline automation guidance for additional details.

4. 📦 Deploy your workload

Without a workload deployed to the cluster it will be hard to see how these decisions come together to work as a reliable application platform for your business. The deployment of this workload would typically follow a CI/CD pattern and may involve even more advanced deployment strategies (such as blue/green). The following steps represent a manual deployment, suitable for illustration purposes of this infrastructure.

• Just like the cluster, there are workload prerequisites to address
• Configure AKS Ingress Controller with Azure Key Vault integration
• Deploy the workload

5. 🏁 Validate

Now that the cluster and the sample workload is deployed; it's time to look at how the cluster is functioning.

• Perform end-to-end deployment validation

🧹 Clean up resources

Most of the Azure resources deployed in the prior steps will incur ongoing charges unless removed.

• Clean up all resources

Preview and additional features

Kubernetes and, by extension, AKS are fast-evolving products. The AKS roadmap shows how quickly the product is changing. This reference implementation does take dependencies on select preview features which the AKS team describes as "Shipped & Improving." The rationale behind that is that many of the preview features stay in that state for only a few months before entering GA. If you are just architecting your cluster today, by the time you're ready for production, there is a good chance that many of the preview features are nearing or will have hit GA.

This implementation will not include every preview feature, but instead only those that add significant value to a general-purpose cluster. There are some additional preview features you may wish to evaluate in preproduction clusters that augment your posture around security, manageability, and so on. As these features come out of preview, this reference implementation may be updated to incorporate them. Consider trying out and providing feedback on the following:

• BYO Kubelet Identity
• Planned maintenance window
• BYO CNI (--network-plugin none)
• Simplified application autoscaling with Kubernetes Event-driven Autoscaling (KEDA) add-on

Related reference implementations

The AKS baseline was used as the foundation for the following additional reference implementations. These build on the learnings of the AKS baseline and applies a specific Lens to the cluster to align a specific topology, requirement, or workload type.

• AKS baseline for multiregion clusters
• AKS baseline for regulated workloads
• AKS baseline for microservices
• Azure landing zones, enterprise-scale reference implementation using Terraform

Advanced topics

This reference implementation intentionally does not cover more advanced scenarios. For example topics like the following are not addressed:

• Cluster lifecycle management with regard to SDLC and GitOps
• Workload SDLC integration (including concepts like Bridge to Kubernetes, advanced deployment techniques, Draft, and so on)
• Container security
• Multiple (related or unrelated) workloads owned by the same team
• Multiple workloads owned by disparate teams (AKS as a shared platform in your organization)
• Cluster-contained state (PV and PVC)
• Windows node pools
• Scale-to-zero node pools and event-based scaling (KEDA)
• Terraform
• dapr

Keep watching this space, as we build out reference implementation guidance on topics such as these. Further guidance delivered will use this baseline AKS implementation as their starting point. If you would like to contribute or suggest a pattern built on this baseline, please get in touch.

Final thoughts

Kubernetes is a very flexible platform, giving infrastructure and application operators many choices to achieve their business and technology objectives. At points along your journey, you will need to consider when to take dependencies on Azure platform features, OSS solutions, support channels, regulatory compliance, and operational processes. We encourage this reference implementation to be the place for you to start architectural conversations within your own team; adapting to your specific requirements, and ultimately delivering a solution that delights your customers.

Related documentation

• Azure Kubernetes Service Documentation
• Microsoft Azure Well-Architected Framework
• Microservices architecture on AKS

Contributions

Please see our Contributor guide.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

With ❤️ from Microsoft Patterns & Practices, Azure Architecture Center.

aks-baseline's Issues

There is no Architecture Diagram or network Diagram available.

Hello Team,

Request you to please provide a high level and low-level architecture diagram for this implementation and also better to provide a deciding factor in design decision eg: why application gateway used here, why used Traefik ingress controller is used?

Just a side note: Network topology diagram which is mentioned in the First page is not available. Please check the below URL:
https://camo.githubusercontent.com/611715e81294cd982921606ca9707d1ee65f4584/68747470733a2f2f646f63732e6d6963726f736f66742e636f6d2f617a7572652f6172636869746563747572652f7265666572656e63652d617263686974656374757265732f636f6e7461696e6572732f616b732f7365637572652d626173656c696e652f696d616765732f7365637572652d626173656c696e652d6172636869746563747572652e706e67

Regards

Azure Policy Blocking Workload

Azure Policy is not admitting the workload due to the following error:

Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [denied by azurepolicy-container-limits-4d458d56ff116b957e42cd9c53b2d7dfd0584f16f78b2df83e21e35a8807f205] container memory limit <256Mi> is higher than the maximum allowed of <512>

We need to fix this so that folks don't have to disable the policy addon to get to the end of this. Might just be a syntax issue between the policy and the limits defined on the workload.

Cannot create service: subset not found

Hi All,

When I trying to connect to run the command below to check the connection, I got the timeout error.

[azureuser@aksbaseline aks-secure-baseline]$ kubectl -n a0008 run -i --rm --tty curl --image=mcr.microsoft.com/powershell --limits=cpu=200m,memory=128M -- curl -kI https://bu0001a0008-00.aks-ingress.contoso.com -w '%{remote_ip}\n'
If you don't see a command prompt, try pressing enter.

curl: (7) Failed to connect to bu0001a0008-00.aks-ingress.contoso.com port 443: Connection timed out
Session ended, resume using 'kubectl attach curl -c curl -i -t' command when the pod is running
pod "curl" deleted

And I got following error when checking the logs of the traefik pod.

[azureuser@aksbaseline aks-secure-baseline]$ kubectl logs -f traefik-ingress-controller-665bc754f9-rh2kt -n a0008
time="2020-09-05T07:31:59Z" level=info msg="Configuration loaded from file: /config/traefik.toml"
{"ingress":"aspnetapp-ingress","level":"error","msg":"Cannot create service: service not found","namespace":"a0008","providerName":"kubernetes","serviceName":"aspnetapp-service","servicePort":"http","time":"2020-09-05T07:33:43Z"}

I am wondering if anyone got this error and how to fix it?

Thanks,
Neal

Do we require cert-manager with Traefik for LetsEncrypt certificates?

Hello Team,

Since Traefik ingress used here out of the box provides HTTPS to microservices by leveraging Let's Encrypt (wildcard certificates support).
Do we also require Cert-manager installed with Traefik ingress just like we do in NGINX?

Align with CAF: Add Reference Tags

The Cloud Adoption Framework highly encourages customers to have specific tags on Azure resources. Let's ensure our baseline is populating tags such as:

• ApplicationName : A0008 (and RegionalHub for hub)
• BusinessUnit : BU0001 (and Networking for hub)
• DR : Essential (and Mission-criticial for hub & spoke)
• Env : Dev (and Prod for hub & spoke)

and of course our standard

• MSPnpRef : aks-baseline
• MSPnpRefVersion : 1.0.0

We can add them as parameters to arm templates, but with these as defaults. That will allow us to update Version number easy if we want. We should also strongly recommend that customers provide their own values for tags, and not just deploy with our defaults.

All suggestions above are just that, suggestions. We can figure out what's best for us.

Why we have WAF in spoke virtual network, when we can have it centrally in Hub Vnet

Hello Team,

In this reference architecture, we are having WAF in Spoke vnet

Can you please explain why to have this in Spoke vnet when we can manage it centrally in Hub Vnet. Any considerations on this?

Regards,

Remove dependencies on public container registries

We currently reference a couple images from quay.io and docker.io. These should be brought into our ACR instance and referenced that way instead. Update the instructions to include this step and changes.

Inner loop script on Bash on VS code with windows machine issue

Opened on behalf of @nitinkhandelwal26

Quote:
While using inner loop script on Bash on VS code with windows machine. I encountered below errors:
Script : aks-secure-baseline/inner-loop-scripts/shell/1-cluster-stamp.sh
Code :

# App Gateway Certificate
openssl req -x509 -nodes -days 365 -newkey rsa:2048
-out appgw.crt
-keyout appgw.key
-subj "/CN=bicycle.contoso.com/O=Contoso Bicycle"
openssl pkcs12 -export -out appgw.pfx -in appgw.crt -inkey appgw.key -passout pass:

Is not working.
I also tried on azure devops windows agent with Bash task, that was also giving same error.

But when i used Ubuntu 18.04 LTS to deploy these scripts, there were working fine, same with Azure DevOps agent.

(Copied from #91)

Unable to fetch object ID

Hi Team,

I was trying to add user in to Azure key vault access policy but I am facing issue .
I am following the below link: https://github.com/mspnp/aks-secure-baseline/blob/main/07-workload-prerequisites.md

My user format is username@customerdomainname

could you please look into this issue.

Subscription functions not consistent

Super nit issue, but I think would help in readability.

Subscription functions are not consistent when creating role definition variables. Some return .id others return .subscriptionId. I've submitted a PR to address.

Examples from template

# Returns .subscriptionId
"acrPullRole": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Authorization/roleDefinitions/7f951dda-4ed3-4680-a7ca-43fe172d538d')]",

# Returns .id
"managedIdentityOperatorRole": "[concat(subscription().Id, '/providers/Microsoft.Authorization/roleDefinitions/f1a07417-d97a-45cb-824c-7a7467783830')]",

Another thing to consider, add a function that takes in the role name (id looking thing) and outputs the resource id for that role. The code could look like this:

Function declaration

"functions": [
        {
            "namespace": "role",
            "members": {
                "resourceId": {
                    "parameters": [
                        {
                            "name": "roleDefinitionId",
                            "type": "string"
                        }
                    ],
                    "output": {
                        "value": "[concat(subscription().Id, '/providers/Microsoft.Authorization/roleDefinitions/', parameters('roleDefinitionId'))]",
                        "type": "string"
                    }
                }
            }
        }
    ]

variable assignment

"networkContributorRole": "[role.resourceId('4d97b98b-1d4f-4787-a291-c67834d212e7')]",
"monitoringMetricsPublisherRole": "[role.resourceId('3913510d-42f4-4e42-8a64-420c390055eb')]",
"acrPullRole": "[role.definition('7f951dda-4ed3-4680-a7ca-43fe172d538d')]",
"managedIdentityOperatorRole": "[role.resourceId('f1a07417-d97a-45cb-824c-7a7467783830')]",
"virtualMachineContributorRole": "[role.resourceId('9980e02c-c2be-4d73-94e8-173b1dc7cf3c')]",
"readerRole": "[role.resourceId('acdd72a7-3385-48ef-bd42-f606fba81ae7')]",

I would be happy to submit the PR if this is of interest.

Service Mesh

Hello Team,

Looking for Guidance on implementing Service Mesh from Infrastructure point.
Do you have any references or roadmap of this implementation?

Regards,

Alert is not generating on P1 after PODFAILED

Hello Team,
we have configured the Alert rule using your code Podfailephase Alert via ARM template cluster stamp.json after that we have configured the Alert with email configuration then we forcefully failed one pod for testing purpose but we are not getting Alert email or any alert notification in portal from podfailedphase alert rule.

could you please help us to figure it out the missing steps.

kubectl get constrainttemplate not returning value (was: AKS managed Internal load balancer is not getting created in cluster version 1.18.8)

Hello Team,

After successful previous deployment in 1.17 v of aks cluster, i tried reploying the cluster with newer templates with version 1.18.8 given in template.
But in new template AKS managed internal loadbalancer is not getting created.

Can you please check this issue.

Helm deployment integration is not available

Hello team, though its not focus on workload, but still its better to show helm deployment integration in combination with Flux.
We are new to AKS, Please give one sample here.

What is XMAS scale out and upgrade Node values in Subnet details table

Hello Team,
Can you please specify what is XMAS Scale-out and Upgrade Node values in the subnet details table mentioned in
https://github.com/mspnp/aks-secure-baseline/blob/main/networking/topology.md

Regards,
Nitin

Prep Baseline for AKS 1.18 (and Ubuntu 18.04)

The baseline is currently targeting 1.17. 1.18 is in preview and also comes with Ubuntu 18.04 by default. Update the reference implementation to support 1.18. We won't plan on merging this this until 1.18 is GA.

Can you provide guidance on Certificate Rotation?

Hi,

Can you please provide some production best practices on refreshing certificate on production system as this is baseline architecture for the production system.
Certificate Rotation is available in Kubernetes from version 1.8, request your secure implementation around this.

Regards,

Not able to access https://bicycle.contoso.com

Deployment is successful, I have created DNS entry but unable to access the website not even by IP.
It throwing 404 error, Any steps to debug the same?

Below is the error:

404 Not Found
Microsoft-Azure-Application-Gateway/v2

There is no azure AAD V2 Documentation available,

Please provide an alternative document for AAD integration because when I click to https://docs.microsoft.com/en-in/azure/aks/managed-aad#before-you-begin
and also this is a preview feature and we don't know when GA of this V2 version will be available.
Request you to provide one sample implementation or documentation till the time AAD V2 is not available.

Upgrade Azure AD Pod Identity bits

Do an upgrade pass on AAD pod identity - to 1.6.3 (or whatever is latest)

• Ensure we're pulling from the new mcr.microsoft.com/oss/azure/aad-pod-identity location as well.

Traefik URL rewriting is not working.

Hello Team,

When implementing Traefik URL rewriting in workload deployment:
https://github.com/mspnp/aks-secure-baseline/blob/main/workload/aspnetapp.yaml

    kubernetes.io/ingress.class: traefik-internal
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.tls.options: default
    traefik.ingress.kubernetes.io/router.middlewares: app-gateway-snet@file, gzip-compress@file
    traefik.ingress.kubernetes.io/rewrite-target: / 
spec:
  # ingressClassName: "traefik-internal"
  tls:
  - hosts:
      - services-00.abc.com
        # In this Implementation we use a
        # wildcard default certificate added at Ingress Controller configuration level which is *.example.com
        # secretName: <services-00-example-com-tls-secret>
  rules:
  - host: services-00.abc.com
    http:
      paths:
      - path: /login

my workload which is having /login path at service, it's not working. can you please help...

Guidance on production AKS cluster upgrade

Hello Team,

Currently, we are using 1.18, and 1.19 will be GA by end of this month.
Any guidance or considerations on the AKS cluster upgrade in a production environment?
Any automation possible using flux on this part?

Cluster-contained state (PVC, etc)

Hello Team,

Any plan to provide reference implementation on statefulset?

Regards,

overlapping service ip range

In your spoke json, it is hard coded for the service cidr range. It seems like a grey area if this ip space can be reused across different clusters? Overlap with other vnets with which the cluster virtual network peers. If spoke A peers with Spoke B, would that be considered overlapping if set on 2 clusters?
https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni

Kubernetes service address range: This is the set of virtual IPs that Kubernetes assigns to internal services in your cluster. You can use any private address range that satisfies the following requirements:

Must not be within the virtual network IP address range of your cluster
Must not overlap with any other virtual networks with which the cluster virtual network peers
Must not overlap with any on-premises IPs
Must not be within the ranges 169.254.0.0/16, 172.30.0.0/16, 172.31.0.0/16, or 192.0.2.0/24
Although it's technically possible to specify a service address range within the same virtual network as your cluster, doing so is not recommended. Unpredictable behavior can result if overlapping IP ranges are used. For more information, see the FAQ section of this article. For more information on Kubernetes services, see Services in the Kubernetes documentation.

Azure Advisor: Still Receiveing PDB Warnings

We had done work to make sure we were not getting any Pod Disruption Budget warnings out of Azure Advisor, but we are still getting them on this. Maybe it's out of our control? If so, we could considering list that on one of the final pages of the readme here so customers are not surprised, maybe where we talk about reviewing Azure Advisor Alerts.

Maybe better, we can see if we can resolve the issue so we don't get the AA alert.

Q: why we require WAF when we have Azure Firewall in hub vnet

Hello Team,

Can you please clarify in this implementation

Why you have used Azure Application Gateway WebApplication Firewall when we have Azure Firewall available in hub vnet,
Cannot we route all incoming traffic to Traefik ingress from Azure firewall using UDR in the Route table?

Request you to please explain What is core logic behind that.

Regards,

Unable to get pod logs from AKS cluster

Hello Team,

Try to get the pod log using :
kubectl logs csi-secrets-store-9wr95 -n cluster-baseline-settings -c secrets-store
and getting below output:
Error from server: Get https://aks-npuser01-42213062-vmss00000c:10250/containerLogs/cluster-baseline-settings/csi-secrets-store-9wr95/s
ecrets-store: dial tcp 10.10.128.197:10250: i/o timeout

Where in portal its showing like

Don't know why I am unable to fetch logs from any pods, Can you please help me here..

Migrate to Azure AD Workload Identities (once User Managed Identity support is in Preview)

Today we install AAD Pod Identity ourselves to make it available to our workloads. AAD Pod Identity is coming as a managed add-on to eliminate this manual step.

1. Remove the .yaml file from the flux configuration
2. Update the workload deployment steps that use AAD Pod Identity to reference the built-in solution (should be no change in instructions though)
3. Update the main page to move that from the OOS list to the features list

VMSS does not support 1.18.2 and 1.18.4

Hi Team,

I am trying to create a cluster with AAD V2 Enable feature but facing below issue.

""The AvailabilityProfile type of agentpool 'npsystem' is set to VirtualMachineScaleSets, but the Kubernetes version is 1.18.4(preview) which doesn't support VMSS."

I have also faced issue while creating cluster with version 1.17 , AKS cluster node pool status was failed.

I am referring link: https://github.com/mspnp/aks-secure-baseline/blob/main/05-aks-cluster.md

Regards,
Rambabu

AAD-V2 out of preview -> Update Readme

Our documentation currently lists the AAD-V2 preview requirement of az feature register --namespace "Microsoft.ContainerService" --name "AAD-V2" (and the subsequent provider registration). That feature is rolling out of preview and should no longer be necessary to do. Once the dust has settled on that, update the readme steps to reflect this.

Can we use Let's encrypt certificate for standard certificate in production deployment

Hi team,

Can we use Let's Encrypt certificate for the standard certificate in production deployment for the ingress controller? or do we need to acquire from a paid CA provider?

For the Azure Application Gateway WAF EV certificate, we will use paid CA certificate providers like entrust.
Q: What is the design consideration for taking an EV certificate here?

Regards,

Github Actions flow has different deployment name

When deploying the cluster using Github actions flow (05 - aks cluster, option 3) the deployment name is cluster-stamp-cd and not cluster-stamp.

This causes one of the following steps to break:

7. Obtain the Azure Key Vault details and give the current user permissions to import certificates.
   KEYVAULT_NAME=$(az deployment group show --resource-group rg-bu0001a0008 -n cluster-stamp --query properties.outputs.keyVaultName.value -o tsv)

Either the deployment name should be changed in GH Action flow, or the doc need to note this.

CSI Driver provider for Keyvault for TLS Cert

https://github.com/mspnp/aks-secure-baseline/blob/main/inner-loop-scripts/shell/1-cluster-stamp.sh#L126-L148
https://github.com/mspnp/aks-secure-baseline/blob/main/workload/traefik.yaml#L281-L286

Why would I want to mount my TLS cert that is only used by my ingress?

Would it not be better to use http://akv2k8s.io to sync the TLS cert to K8s?

Upgrade Azure Key Vault provider for Secret Store CSI driver to 0.0.9 (from 0.0.6)

At the time this was opened, 0.0.9 is the current production version. But when this task is picked up evaulate any newer version as well.

Azure Policy assignments are not getting deployed into the AKS cluster

it is being reported that the kubectl get constrainttemplate does not return any resource so we want to clarify what would be the "correct" response to that command.
Originally posted by @ckittel in #13 (comment)

Update Azure AD AddOn to 1.7.0

Latest Azure AD Addon has been released, upgrade to 1.7.0 (or whatever is newer once this is picked up)

https://github.com/Azure/aad-pod-identity/releases/tag/v1.7.0

base64 -w is not available on macOS

This doc includes instructions for base64 encoding certificates using bas64 -w. The -w option is not available on macOS and would rather be -b 0 which is also the command default so can be omitted.

https://github.com/mspnp/aks-secure-baseline/blob/main/02-ca-certificates.md

Q: What are the considerations in choosing Traefik ingress controller

Hi Team,

I have several points which are not clear/explained from the documentation available at aka.ms/architecture/aks-baseline:

Q What are the considerations you took for this case study in choosing the Traefik ingress controller with Azure Application Gateway when we have AGIC available which have integrated WAF?.
Q: Why you defined workload-level ingress rather than cluster-level ingress? if we define cluster-level ingress then any special considerations do we need to make? - require guidance.

Regards,

Warning in CSI logs

Hello Team,

I just encountered one warning in CSI logs
kubectl logs csi-secrets-store-9wr95 -n cluster-baseline-settings -c secrets-store

Though it's not an issue and doesn't know how it will impact the CSI store. just to bring into your notice..

Generate a globally unique Log Analytics resource name for hub-spoke network

The LA name for the network template is not globally unique. Generate one in the default hub that can be also infered from the spoke. If it cannot be infered, then we'll need to pass that information into the spoke -- but let's try to make a unique string that uses the subscription+resource group+region for the regional hub.

AKS cluster not visible while logging using AD user bu0001a0008-admin

Hi Team,

I have deployed the full solution in my environment but I am not able to see the cluster if I login using AD user as bu0001a0008-admin .

could you please suggest how can i get access or any additional configuration needs to be done.

after add IAM role manually , i am able to access the cluster.

K8s AAD Group -> Azure RBAC

Create a branch that replaces the k8s AAD Group based auth to Azure RBAC mapped access. This shouldn't land until GA, but we can have it tested and ready to go ahead of GA.

Upgrade Cluster to AKS 1.19

Ensure the following configuration is validated and accounted for:

• ContainerD
• Default Custom Node Config placeholder
• Ephemeral OS disk
• AutoUpgrade = False (if available in API by that time)

Consider updating location default value to use dynamic expression

The location parameter default value is set as eastus2. Consider updating with the expression [resourcegroup().location].

https://github.com/mspnp/aks-secure-baseline/blob/main/cluster-stamp.json#L43

Understood that the allowed values are limited to availability set enable regions, I can confirm that the ARM validation API will evaluate the expression and enforce the allowed values.

[Azure Policy] Restrictive Initiative showing out of compliance

We have this initiative set to Audit, and we have four failures in our cluster. Can we clean up our exclusions or policy defination to bring us back in line with our expected stance?

Unable to Access Application

Hi Team,

I have deployed cluster with workload successfully but I am unable to access application running on cluster.
I have followed the below link:
https://github.com/mspnp/aks-secure-baseline/blob/main/10-validation.md

please find the Attached screenshot.

Deployment failing with below error.

Deployment failed. Correlation ID: b6887f5f-84ef-4374-ac6e-b25a25b57392. {
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'.",
"details": [
{
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n "error": {\r\n "code": "RoleAssignmentUpdateNotPermitted",\r\n "message": "Tenant ID, application ID, principal ID, and scope are not allowed to be updated."\r\n }\r\n}"
}
]
}
]
}
}

Deployment Issue: Principal Not Found

I am experiencing a high deployment failure rate due to principal not found during deployment. Sometime the deployment is successful, other times it fails. After a failure, I can re-deploy the tempalte, which will then succeed.

I know that we once had 'replication' latency on managed service identities, which would cause similar issues. However as I understand it, this was fixed with the role assignment resource of version 2018-09-01-preview which is being used here. ARM tempalte dependencies are in place and look good.

I wonder if this is isolated to myself or if others are seeing similar issues?

set -e does not work in inner-loop-scripts

There are several "kubectl --watch" command in the script 1-cluster-stamp.sh and "set -e" will break the script with Ctrl-C.

Azure API Management in between Azure App Gateway and ingress controller

Hello Team,

If our cost model is to charge the customer based on API incoming requests or the number of hits then would it be wise to include API management between App Gateway and Ingress controller? or any other components that can be used for this purpose? Request you to please provide Guidance or Suggestions on this.

Regards,