Compiled Version? about adcs-issuer

[eyolas]

hi @chojnack, any news ?

[Antebios]

@xargyx HOLY CRAP, IT WORKED!!!!

To let everyone know the image name is "xargyx/adcs-issuer:v0.3". Within template.yaml, under the Deployment component, I set the container image to the image above (as well as setting the nodeSelector to kubernetes.io/os: linux because I have a Windows worker node in my cluster), and then added another argument: - --adcsTemplateName=WebServer.

I then applied my certificate request from my previous post: #1 (comment) and I can see that the certificate request is approve, created, and the tls.crt and tls.key are created. I also decrypted the tls.crt to verify the certificate. It's working!!!

Thanks to @ctrought and everyone else who has helped me.

My environment consists of the latest kubernetes v1.22.1, cert-manager v1.5.3, and Windows Server Core 2016 as my domain controller/dns running Windows Active Directory Certificate Services. I am using @ctrought template.yaml with @xargyx docker image and config changes.

[xargyx]

@Antebios in the issuer.go file the template is populated with

adcsCertTemplate = "BasicSSLWebServer"

I took the changes here

SimeonPoot@9ba12af

and built my own image with the new repo and added those changes to make the template configurable. If you're looking for the ability to make the template name configurable.

[KodoGit]

Create AD user and give it permissions to enroll the template.
The template should accept subject name from request.

Install cert-manager

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml
kubectl get ns| grep cert-manager

Make a working folder and setup selfsigning cluster issuer. Naming is important.

mkdir autocert && cd autocert
nano cluster-self.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

kubectl apply -f cluster-self.yaml

Get pearj/adcs-issuer repo and make changes

mkdir git-pearj && cd git-pearj
git clone https://github.com/pearj/adcs-issuer
cd adcs-issuer
sed -i "s/image: controller:latest/image: ghcr.io\/pearj\/adcs-issuer:v1.7-beta.4/g" config/manager/manager.yaml

#Only if you have and old kubectl version and get errors applying config/default
#nano config/default/kustomization.yaml
#Change resources: -> bases: at line 15

Start controller

kubectl apply -k config/default

Check that you got pods running and tech cert READY:

kubectl get pods -n adcs-issuer-system
kubectl get cert -n adcs-issuer-system

Put AD user credentials as plain text to secret.

cd ../../
nano ca-secret.yaml

apiVersion: v1
stringData:
  password: PASSWORD
  username: SAMACCOUNTNAME
kind: Secret
metadata:
  name: adcs-credentials
type: Opaque

Apply it to adcs-issuer-system namespace.
#Looks like default Nokia's version searched the secret in kube-system namespace.

kubectl apply -f ca-secret.yaml -n adcs-issuer-system

Get your CA web cert for caBundle. Change ca.example.com to your host.

echo QUIT|openssl s_client -showcerts ca.example.com:443 2>/dev/null </dev/null |  sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'| base64 -w 0

Create Issuer yaml. Edit caBundle, url, templateName.

nano ClusterAdcsIssuer.yaml

apiVersion: adcs.certmanager.csf.nokia.com/v1
kind: ClusterAdcsIssuer
metadata:
  name: cluster-adcs-issuer
spec:
  caBundle: PUT_THE_HUGE_STRING_HERE
  credentialsRef:
    name: adcs-credentials
  statusCheckInterval: 6h
  retryInterval: 1h
  url: https://ca.example.com/certsrv/
  templateName: WebServerSHA2

kubectl apply -f ClusterAdcsIssuer.yaml

Make a test cert

nano testcert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  annotations:
  name: testcert
  namespace: default
spec:
  commonName: testcert.example.com
  dnsNames:
  - testcert.example.com
  issuerRef:
    group: adcs.certmanager.csf.nokia.com
    kind: ClusterAdcsIssuer
    name: cluster-adcs-issuer
  secretName: testcert

kubectl apply -f testcert.yaml
kubectl get cert testcert

If the cert is not READY, look at CA failed requests logs (CA manager mmc) or Kube logs:

kubectl logs $(kubectl get pod -n adcs-issuer-system|tail -n 1| cut -d ' ' -f 1) -n adcs-issuer-system manager| tail -n 1

To rebuild the cert again

kubectl delete cert testcert && kubectl apply -f testcert.yaml

[ctrought]

@ctrought Thank you for the template and container image! I had to update the template to get it installed in my k8s 1.22.1 cluster. I also discovered that I needed to have the "certificate.yaml" applied to the adcs-issuer namespace first before the Pod would start up. Now I need to create an Issuer for my Active Directory Certificate Server, wish me luck!

No problem, glad it helped! If you have trouble with the image let me know, I believe that image was built based an older version of certmanager which is mentioned in the (now) outdated readme.

ADCS Issuer has been tested with cert-manager v.0.11.0 and currently supports CertificateRequest CRD API version v1alpha2 only.

There are some kubernetes apis that were in these versions that were deprecated and are now removed in 1.22 which means that build I have likely won't work on 1.22, and I am guessing thats also probably why you had to update the template (beta webhook + admission apis were part of those removed in 1.22 as they are now v1) you can read about that here.

https://cert-manager.io/docs/release-notes/release-notes-1.5/#api-deprecation

There have been some commits by some other helpful people in this repo to bring it more up to date and leverage cert-manager 1.3+ and the v1 api, so I will likely rebuild and push a new version in case anyone has trouble building themselves.

[Antebios]

@ctrought I'm sorry, but I submitted that response by mistake. I had set my login User Name incorrectly. I corrected it and now I can authenticate the ADCS website and seeing in the log:
1 ntlm_certsrv.go:316] Getting CA Chain from ADCS Certsrv http://macross-dc.macross.com/certsrv/

The status message I see in the CertificateRequest object is ADCS request rejected. I attempted to submit a certificate manually using the ADCS website and I received the error message:
Your Request Id is 6. The disposition message is "Error Parsing Request ASN1 bad tag value met. 0x8009310b (ASN: 267 CRYPT_E_ASN1_BADTAG)".

I suspect it might be because I had to remove the Organization attribute from the Certificate Request since cert-manager.io/v1 did not like it. But I am finally making good progress, thanks to you!

[Antebios]

@ctrought There is a tiny mistake in your template.yaml file. There needs to be a "---" separator after line # 267. It needs to separate the CRD from the Namespace. Applying it was failing for me until I figured it out.

status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
apiVersion: v1
kind: Namespace
metadata:
  labels:
    control-plane: controller-manager
  name: adcs-issuer-system
---

[xargyx]

@ctrought thank you for the work you put in! I'm able to get certificates now on the most recent cert manager.

@Antebios I did put it on dockerhub, all you gotta do is add the argument (line 825 on the template.yaml)

• --adcsTemplateName=

[lknite]

@pearj Might you be interested in updating the getting started instructions at your https://github.com/pearj/adcs-issuer site?

[lknite]

This is way more difficult to install than it should be. We need a helm chart or something.

[makeacode]

caveat...still learning about CRDs...I now realize the CRDs are actually in the config/crds path as the installation section states...but I would still need the image built

[chojnack]

Hi,
We're working currently on making adcs-issuer compatibile with newest cert-manager release. We'll also try to create a builded and easily installable release soon.
Thank you.

Marcin Chojnacki

[makeacode]

@chojnack Thanks fo rhte response...looking forward to testing this out soon. I'll probably experiment with the current release and see what I can get going...if anything.

[ctrought]

Also interested in a compiled version 👍

[BatmaniNRobin]

Interested in this as well ! 🙏

[ctrought]

Interested in this as well ! 🙏

I ended up building with kubebuilder as per their docs. The template it generates is available here, I have pushed the controller image to docker.io as well.

https://github.com/ctrought/adcs-issuer/blob/master/template.yaml
https://hub.docker.com/r/ctrought/adcs-issuer

[Antebios]

@ctrought Thank you for the template and container image! I had to update the template to get it installed in my k8s 1.22.1 cluster. I also discovered that I needed to have the "certificate.yaml" applied to the adcs-issuer namespace first before the Pod would start up. Now I need to create an Issuer for my Active Directory Certificate Server, wish me luck!

[Antebios]

@ctrought Thank you for the feedback. I did use that documentation link to update the configuration file. I think I will absolutely like a newer build of the controller. It is failing to submit my CSR to my ADCS server because it is failing to find the Issuer Credentials, but I know it's all there in the same namespace.
Secret \"test-adcs-issuer-credentials\" not found"

If I get this up and running successfully I hope to document my journey in some sort of guide in both here in GitHub and Medium.

[ctrought]

@ctrought Thank you for the feedback. I did use that documentation link to update the configuration file. I think I will absolutely like a newer build of the controller. It is failing to submit my CSR to my ADCS server because it is failing to find the Issuer Credentials, but I know it's all there in the same namespace.
Secret \"test-adcs-issuer-credentials\" not found"

If I get this up and running successfully I hope to document my journey in some sort of guide in both here in GitHub and Medium.

There is a flag that is used to set this in the controller-manager pod args, if not set it uses kube-system to look for cluster resources including the secret. I ran into this issue too, it was quite a mystery to me :) So you can either add the flag to the deployment arg or move the secret to kube-system.

The code

flag.StringVar(&clusterResourceNamespace, "cluster-resource-namespace", "kube-system", "Namespace where cluster-level resources are stored.")

Example

spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: controller-manager
  template:
    metadata:
      labels:
        control-plane: controller-manager
    spec:
      containers:
      - args:
        - --metrics-addr=127.0.0.1:8080
        - --enable-leader-election
        - --cluster-resource-namespace="adcs-installed-namespace"

[ctrought]

@ctrought Thank for your help thus far, my error message is resolved. I finally have it attempting to submit a CSR, but now I am being challenged with authentication:

<title>401 - Unauthorized: Access is denied due to invalid credentials.</title>

It is obviously the credentials.
apiVersion: v1 data: password: somepassword # base64 encoded username: MyDomain\LogonName # base64 encoded kind: Secret metadata: name: test-adcs-issuer-credentials namespace: adcs-issuer-system type: Opaque

What username and password should I be using? I am using my domain's logon and password, and I can see them clearly in the k8s dashboard. I am sooooo close.

Indeed you look very close! Unfortunately I can't really say for sure what credentials you should be using. In my case, I don't need to provide the domain with my username but I don't know whether that would be the same for you. Have you/are you able to try authenticating via web browser to verify that your ADCS is accepting those credentials?

[karezza]

@ctrought I am also interested in an updated image

Given a note you posted before, maybe things are not going to work because I'm trying to use the latest and greatest cert-manager version:

ADCS Issuer has been tested with cert-manager v.0.11.0 and currently supports CertificateRequest CRD API version v1alpha2 only.

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
...

$ k apply -f template.yaml 
...
unable to recognize "template.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "template.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "template.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "template.yaml": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "template.yaml": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"

$ k version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:39:34Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}

[Antebios]

@karezza I also got those errors when applying the "template.yaml", so I had to manually update the failed components so they would work with k8s v1.22.x. It took me trial and error, but I did it. Now, did I do it correctly? I don't know. Everything looks like I did and you can see the challenges I had/have above.

[ctrought]

@ctrought I am also interested in an updated image

Given a note you posted before, maybe things are not going to work because I'm trying to use the latest and greatest cert-manager version:

ADCS Issuer has been tested with cert-manager v.0.11.0 and currently supports CertificateRequest CRD API version v1alpha2 only.

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
...

$ k apply -f template.yaml 
...
unable to recognize "template.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "template.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "template.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "template.yaml": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "template.yaml": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"

$ k version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:39:34Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}

I will post the latest compiled version and updated template next week (away from laptop at the moment).

[ctrought]

@Antebios @karezza I have pushed a newer build, tagged 0.3. The generated template.yaml assumes cert manager is installed under the cert-manager namespace. https://hub.docker.com/repository/docker/ctrought/adcs-issuer

[karezza]

@ctrought @Antebios, I've cloned ctrought's repository & followed the steps at the https://book.kubebuilder.io/quick-start.html#installation site. Some quick questions:

• I end up with a pod adcs-issuer-controller-manager looking for a secret webhook-server-cert, but I don't see this in the instructions, do I need to set this up (or does this not matter if I set up to auto approve?)
• The example issuer has an attribute "url: " what is that url, or again, does this not matter if it is setup to auto approve?
• caBundle: It's not obvious to me what this should be based on the instructions, it sounds like this is specific to the ADCS controller, ad not the actual ca, initially I would have thought this would be the public key of the ca ... rather than guess, what is it expecting?

Events:
  Type     Reason       Age                   From     Message
  ----     ------       ----                  ----     -------
  Warning  FailedMount  52m (x4 over 77m)     kubelet  Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[kube-api-access-77cdj cert]: timed out waiting for the condition
  Warning  FailedMount  7m30s (x28 over 86m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-77cdj]: timed out waiting for the condition
  Warning  FailedMount  3m22s (x50 over 88m)  kubelet  MountVolume.SetUp failed for volume "cert" : secret "webhook-server-cert" not found

[ctrought]

@karezza

I end up with a pod adcs-issuer-controller-manager looking for a secret webhook-server-cert, but I don't see this in the instructions, do I need to set this up (or does this not matter if I set up to auto approve?)

This secret should contain a cert for the webhook service. In my case I am using OpenShift, so I am simply annotating the service and the cluster network operator takes care of creating the certificate signed by the internal CA.

For others that might mean using cert-manager to create an internally signed certificate for the adcs-issuer-webhook-service, you can find some examples here and it shows the annotations you'd need to add to the Mutating/Validating webhook CR's to have cert-manager inject the CA config into it (actually there are a bunch of lines in the kustomization.yaml configs that are commented, that if uncommented will do all this for you, see config/certmanager/certificate.yaml, config/crd/kustomization.yaml,config/default/kustomization.yaml When kubernetes calls the webhooks, it needs to be aware of the CA in order to verify the webhooks certificate.

You might even use the kubernetes provided api to do this if you didn't want to use cert-manager internal signer for some reason.

The example issuer has an attribute "url: " what is that url, or again, does this not matter if it is setup to auto approve?

If you're trying configuring an ADCS issuer, you will need to provide the URL for the ADCS endpoint that issues the certificates. It would be some external endpoint like this https://<internal-host>.ca/certsrv

caBundle: It's not obvious to me what this should be based on the instructions, it sounds like this is specific to the ADCS controller, ad not the actual ca, initially I would have thought this would be the public key of the ca ... rather than guess, what is it expecting?

This has to do with the ADCS url as it will be signed by some certificate authority, so we need to provide that to ADCS issuer so it can verify the cert during it's API requests when retrieving certs.

ps: if you had any specific errors from the template that I stuck in my fork, would you be able to open an issue there or let me know any errors you encountered relating to the spec?

[lknite]

It's not clear to me what to uncomment or not. Are you thinking to put together a helm chart and keep this going? Your efforts would certainly be appreciated.

[xargyx]

I got the new adcs-issuer working with the latest cert-manager and it produces a valid certificate (stored in the tls cert field). However the ca cert that is stored inside the the certificate is incorrect (idk what value it's getting now). It used to get the adcs issuer certificate in that field (same value in the caBundle field). Not exactly sure if the adcs-issuer or cert manager stores that field. If i manually edit the certificate and put the adcs issuer cert in that field the cert works fine.

I didn't see anything in the code changes that would impact this, so I'm thinking it must be something different in cert-manager.

[Antebios]

@ctrought I was finally able to put time aside to try to get this working. I was excited, but my hopes were dashed.

My test AdcsIssuer (cert-manager.io --> Certificate --> test-adcs) has a message that says:
Issuing certificate as Secret does not exist

There was a secret created within the "adcs-issuer-system" namespace named after my test certificate request, "adcs-cert", that contains a valid tls.key

The adcs-issuer-controller-manager pod --> "manager container is throwing this error message:
1 leaderelection.go:329] error initially creating leader election record: leases.coordination.k8s.io is forbidden: User "system:serviceaccount:adcs-issuer-system:default" cannot create resource "leases" in API group "coordination.k8s.io" in the namespace "adcs-issuer-system"

[ctrought]

cannot create resource "leases" in API group "coordination.k8s.io" in the namespace "adcs-issuer-system

Sorry, I had encountered this a while ago and committed changes to a private repo I mainly use but not the public one. The template has been updated which simply adds the create verb for leases to the role.

As for the message about the secret, this is generated by cert-manager when it detects the secret referenced by the Certificate resource to store the issued key/cert does not exist. It would be looking in the same namespace where the Certificate resource exists, which is not necessarily adcs-issuer-system. It is more informational though, as it needs to have adcs-issuer fetch the cert from ADCS and then save to the secret.

[Antebios]

@ctrought Thanks for the response. I updated my template.yaml and then re-created by certificate request with no other changes... and it worked and failed at the same time.

It worked because it was able to communicate with my AD Certificate Server. And it failed because the certificate request was denied because the template being submitted wasn't correct.

status:
  id: '9'
  reason: >-
    Denied by Policy Module  0x80094800, The request was for a certificate
    template that is not supported by the Active Directory Certificate Services
    policy: BasicSSLWebServer. The requested certificate template is not
    supported by this CA. 0x80094800 (-2146875392
    CERTSRV_E_UNSUPPORTED_CERT_TYPE)
  state: rejected

Here is my certificate request (don't worry, this is my internal homelab domain):

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  annotations:
  name: adcs-cert
  namespace: adcs-issuer-system
spec:
  commonName: smartconnect.macross.com
  dnsNames:
  - smartconnect
  - smartconnect.macross.com
  issuerRef:
    group: adcs.certmanager.csf.nokia.com
    kind: AdcsIssuer
    name: test-adcs
  #organization:
  #- MPI
  secretName: adcs-cert

I'm almost there... I can feel it!

[Antebios]

@xargyx Is your adcs controller docker image on docker hub? Otherwise, I need to create my own? What yaml change(s) do I need to make for my certificate request to set the template to use?

@ctrought Are you going to make the same changes?

[lknite]

@Antebios I'm trying to get the same use case working as you did. Is this the template file that you used? https://github.com/ctrought/adcs-issuer/blob/master/template.yaml

I'm seeing an error saying a secret is not found when I test with a certificate.yaml, but I can definitely see the secret in the specified namespace, was there another step in your setup?

$ k -n adcs-issuer-system logs -f adcs-issuer-controller-manager-56cb6b4fdb-lj775 manager
{
	"level": "error",
	"ts": 1636669721.3535452,
	"logger": "controller-runtime.manager.controller.adcsrequest",
	"msg": "Reconciler error",
	"reconciler group": "adcs.certmanager.csf.nokia.com",
	"reconciler kind": "AdcsRequest",
	"name": "adcs-cert-pkp75",
	"namespace": "adcs-issuer-system",
	"error": "Secret \"adcs-issuer-credentials\" not found",
	"stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214"
}

$ k -n adcs-issuer-system get secrets adcs-issuer-credentials 
NAME                      TYPE     DATA   AGE
adcs-issuer-credentials   Opaque   2      67m

[Antebios]

@lknite Yes, that is the same template. I place my "adcs-issuer-credentials" secret in both namespaces "kube-system" AND "adcs-issuer-system". The format of the secret username="DOMAIN\login" and password="loginPassword".

[lknite]

@Antebios Thank you! That got me past the secret not found error. Now running into a 401 error, can't imagine this will be too hard to figure out. Troubleshooting... Thanks again.

[lknite]

I'm running into the following Access is denied error:

  <h2>401 - Unauthorized: Access is denied due to invalid credentials.</h2>
  <h3>You do not have permission to view this directory or page using the credentials that you supplied.</h3>

I can browse to the following url and login so I know the credentials are good:

https://<mydomain>/certsrv/

Made sure to use base64 encoded string 'mydomain\username' in my secret. Also tried the following technique:

https://www.admin-enclave.com/en/articles/windows/129-solve-the-error-401-when-you-open-the-certsrv-from-a-computer.html#:~:text=401%20%2D%20Unauthorized%3A%20Access%20is%20denied,the%20credentials%20that%20you%20supplied.

The technique above still results in a 401 (I did remember to restart iis). I am using Windows Server 2019, maybe this doesn't work with Windows Server 2019? Anyone have success using Windows Server 2019? If not, I have a powershell script we could use that works. If we went with PowerShell we could use modern security techniques. (using Powershell module: PSPKI)

There seem to be several local security policy / default domain policy values around NTLM. Could someone share which worked for them?

[pearj]

Hi @lknite,

I'm using Server 2019 and it worked fine for me.

I didn't prefix the username with mydomain\.

You don't have to base64 encode the string if you use stringData when creating the secret, eg:

apiVersion: v1
stringData:
  password: "cleartext password"
  username: "cleartext username"
kind: Secret
metadata:
  name: test-adcs-issuer-credentials
  namespace: <namespace>
type: Opaque

It will automatically base64 encode the values for you when it is created.

I didn't setup ADCS so I don't know if we did anything special with the group policy relating to NTLM.

For what it's worth I'm using my own version of the adcs-issuer:
https://github.com/pearj/adcs-issuer

Which is a merge of:

https://github.com/UKCloud/adcs-issuer
and
https://github.com/ctrought/adcs-issuer

I also added the added the ability to specify which template to use in the AdcsIssuer/ClusterAdcsIssuer.

The docker images are published straight to the GitHub container registry: https://github.com/pearj/adcs-issuer/pkgs/container/adcs-issuer.

[KodoGit]

Hi. How can I install this issuer?
I've cloned @pearj repo and run `make install'
'make run' fails with "cache not synced" errors
If I apply template.yaml the containers are not created

Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    6m29s                  default-scheduler  Successfully assigned adcs-issuer-system/adcs-issuer-controller-manager-5d655dc55f-m8zgf to certmanager-control-plane
  Warning  FailedMount  2m10s (x2 over 4m26s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert default-token-rnkb7]: timed out waiting for the condition
  Warning  FailedMount  17s (x11 over 6m29s)   kubelet            MountVolume.SetUp failed for volume "cert" : secret "webhook-server-cert" not found
Error from server (NotFound): pods "kube-rbac-proxy" not found

[pearj]

Hi @KodoGit,

I actually never tried make run or make install, so it's quite possible they're broken. I only tried make on it's own.

Also I don't use template.yaml either, so it also is probably missing some of the changes to the CRD I made.

I guess I should have edited the README.md a bit more!

I only use the kustomize template inside ArgoCD with some image overrides. Here is my ArgoCD application manifest:

project: default
source:
  repoURL: 'https://github.com/pearj/adcs-issuer.git'
  path: config/default
  targetRevision: HEAD
  kustomize:
    images:
      - >-
        gcr.io/kubebuilder/kube-rbac-proxy=gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0
      - >-
        controller=ghcr.io/pearj/adcs-issuer:v1.7-beta.4
destination:
  server: 'https://kubernetes.default.svc'
  namespace: adcs-issuer-system

The main thing is that you need to override the controller image when using my kustomize template, as I leave the controller image as just controller which won't resolve on docker hub. So you could use my automatically built image that is published to the github container registry: ghcr.io/pearj/adcs-issuer:v1.7-beta.4

So if you edit the manager.yaml and replace controller:latest with ghcr.io/pearj/adcs-issuer:v1.7-beta.4

Then using kubectl you should be able to run kubectl apply -k config/default.

If you want to see how I build and create the docker image look at ci.yaml as that is what I test.

[KodoGit]

Thanks for reply. But I started randomly pressing buttons in kubernetes like yesterday and most of your post looks like black magic :) Maybe I should look at some "hashi vault"

[pearj]

@KodoGit if you're just new to Kubernetes, you really have jumped in at the deep end by trying this!

If you just wanted to try deploying using just kubectl then just doing these 2 steps will probably getting you an empty adcs-issuer.

So if you edit the manager.yaml and replace controller:latest with ghcr.io/pearj/adcs-issuer:v1.7-beta.4

Then using kubectl you should be able to run kubectl apply -k config/default.

Can't really comment on whether hashicorp vault is better for you use-case or not. I was going to go down a similar path, but we needed ADCS in our environment for other Windows reasons, so it made sense for use to have just one CA and not multiple.

[KodoGit]

@pearj thx. It's alive! And issuing.
I had to add cluster issuer "selfsigned". Seems like errors I've posted above were because of that.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

[lknite]

@KodoGit I wonder, before your mind moves too far on to other things, if you might consider writing up a fairly detailed how-to guide. The guide on this project is out of date and the details in this thread seem to have everything but given that multiple people asked and were help the flow the question/answer is a bit all over the place. I would write a guide myself but I have yet to get this to work.

[lknite]

Hello, attempting this again from scratch as I have reinstalled my systems and and clusters.

1. Followed the steps provided above marked, "Antebios commented on Oct 22, 2021", applying the template.
2. Created a tls secret called webhook-server-cert
3. applied a ClusterAdcsIssuer
4. created two secrets 'adcs-issuer-credentials' in both namespaces 'kube-system' and 'adcs-issuer-credentials'

I'm seeing this in the manager log:

level      : error
ts         : 1642910529.73091
logger     : setup
msg        : problem running manager
error      : failed to wait for certificaterequest caches to sync: no matches for kind "CertificateRequest" in version "cert-manager.io/v1"
stacktrace : main.main
                /home/runner/work/adcs-issuer/adcs-issuer/main.go:161
             runtime.main
                /opt/hostedtoolcache/go/1.16.12/x64/src/runtime/proc.go:225

Next I installed 'cert-manager' via bitnami, wasn't sure if I needed to install that in addition to the adcs-issuer. Uninstalled and reinstalled adcs-issuer. Still getting same result.

Did I miss a step? Or is there something else to check?

[KodoGit]

@lknite, hi. I'm going to rebuild my test lab and write step by step script this week. Your error looks like mine when I did not have ClusterIssuer for self signing certs. Look at my last yaml.

[lknite]

@KodoGit thanks, look forward to it.

Last time I installed via cert-manager repo, this time I used a helm install via bitnami. Only just figured out that I needed to add "--set installCRDs=true" to my helm install. Things look to be working better now. Troubleshooting ...

Back to getting a 401 like last time. Verified password is correct like before.

$ cat adcs-issuer-credentials.yaml 
apiVersion: v1
data:
  username: <base64 encoded username>
  password: <base64 encoded password>
kind: Secret
metadata:
  name: adcs-issuer-credentials
  namespace: adcs-issuer-system
type: Opaque

 cat adcs-issuer-credentials_kube-system.yaml 
apiVersion: v1
data:
  username: <base64 encoded username>
  password: <base64 encoded password>
kind: Secret
metadata:
  name: adcs-issuer-credentials
  namespace: kube-system
type: Opaque

Am able to log on using the same credential via https://domain/certsrv when browsing to the site by hand.

[lknite]

These instructions are really good. I reviewed them and started looking to how I could contribute. I started looking up powershell commands to install a cert authority and then for a command to add a user to have access to issue certificates. Turns out I had created a user but had not yet added them to have permission to issue certificates. I had performed this step with my last attempt, but not this time. Now things worked for the first time. A certificate was issued via annotations in an ingress. Though, when I browse to the site its still showing as the fake kubernetes, this is close. I can see the secret which was created is not using the same name as I specified the secretName would be in the ingress. Troubleshooting...

Maybe a page with break/fixes would also be good? Also, maybe a page of which versions of which things are working together nicely. I'm still using 1.6.1 of cert-manager, going to see if that will work. Will update here with more details of a working scenario once I get past this last hurdle.

[lknite]

I'm back to getting a 401 error this morning. Will wipe everything out and retry your directions exactly soon as I have a chance.

[pearj]

@lknite Sorry for the slow response. I'm happy to accept PR's on my repo. Or maybe I can give permission for you both to edit the wiki?

[lknite]

@pearj could you enable issues on your github site so we can chat a bit there?

[lknite]

@KodoGit could you possibly do a pull request with your instructions on perlj's page? https://github.com/firstcontributions/first-contributions

Also, a request, in the instructions when you specify a yaml could you be sure to also specify a namespace (where one can be specified)?

[lknite]

@KodoGit when I 'k apply -f config/default' I get these validation errors:

$ k apply -f config/default
mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating-webhook-configuration unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webhook-configuration unchanged
error validating "config/default/kustomization.yaml": error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with --validate=false
error validating "config/default/manager_auth_proxy_patch.yaml": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false
error validating "config/default/manager_prometheus_metrics_patch.yaml": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false
error validating "config/default/manager_webhook_patch.yaml": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false

Currently running in a cluster v1.18

[KodoGit]

@lknite I'm not sure about which Kustomise version is used by your kubectl and when "resources" appeared first.
Try to change "resources" to "bases" as my comment about applying of config/default errors says. Right at that manual.

[KodoGit]

specify a namespace

You don't need to specify any namespaces as this story is about cluster wide issuers. Just follow the instruction as it is.
After test cert is successfully issued in default NS you can use testcert.yaml as a template for your own certs yamls for other NS.

[lknite]

@KodoGit Your instructions worked well, and were fast and easy to follow. Nice one liner to get the caBundle!

Had to make the following adjustments:

- use 'https://<host>/certsrv' rather than 'https://<host>.<domain>/certsrv' in ClusterAdcsIssuer.yaml
- add an environment variable to work with just the hostname:
        env:
        - name: GODEBUG
          value: x509ignoreCN=0
- change 'templateName: WebServer' rather than 'templateName: WebServerSHA2' in ClusterAdcsIssuer.yaml

Feel like I'm really close. Currently seeing the following error:

$ k get adcsrequests.adcs.certmanager.csf.nokia.com -n demo demo.k-dev-br4pd -o yaml
...
Denied by Policy Module The permissions on the certificate template do not
    allow the current user to enroll for this type of certificate. 0x80094012 (-2146877422
    CERTSRV_E_TEMPLATE_DENIED)

After creating an ad user for the adcs-issuer to use I opened up the Certification Authority msc, right-clicked and selected properties, clicked the Security tab, then added the ad user and granted them all 4 rights available. Something else needed to grant the needed rights?

[lknite]

Found the answer right after posting. There was an additional step to configure the template to allow the ad user I created to have access. The steps at this link worked for me: https://docs.microsoft.com/en-us/windows-server/networking/core-network-guide/cncg/server-certs/configure-the-server-certificate-template

I can report success following KodoGit's instructions with the changes in the previous post and the additional permissions in this post. Kubernetes version 1.22.4, cert-manager 1.5.3, adcs-issuer provided by perlj v1.7-beta.4

** update: also need to create secret webhook-server-cert, and create secret adcs-issuer-credentials in the kube-system namespace

[pearj]

@pearj could you enable issues on your github site so we can chat a bit there?

Ok that’s done.

[BatmaniNRobin]

After retrying and following along with what @KodoGit laid out alongside applying the template.yaml, I was able to get everything working and deployed; however, it requires the manual deletion of the ca.crt which is auto-generated whenever any certificate is created. (Current working theory is that the caBundle is incorrect causing the ca.crt to be invalid and preventing nginx from using the certificate also because it is the only component of this that i'm not able to confirm is 100% correct).
Diagram of system is 1 root node with two intermediary nodes, adcs-issuer is only pointing at 1 inter node. the caBundle kodogit creates was tested and so was the ca certificate chain.p7b on ca.example.com/certsrv/.

The error that seems to persist within the nginx ingress pod logs via:
kubectl logs pod/ingress-nginx-controller-<HASH_OF_YOUR_POD> -n ingress-nginx | grep cert
is:
backend_ssl.go:46] Error obtaining X.509 certificate: parsing CA certificate: asn1: structure error: tags don't match (16 vs {class:0 tag:6 length:9 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} tbsCertificate @2 7 controller.go:1196] Error getting SSL certificate "adcs-issuer-system/adcs-cert-auto-generate": local SSL certificate adcs-issuer-system/adcs-cert-auto-generate was not found. Using default certificate

Passing this certificate into an online decoder also says that the format is invalid or corrupted and is thus unable to decode but unsure how to go about fixing this.
cert-manager says that ca.crt shouldnt be stored in the same secret if the CA is known and so does that mean that ADCS-issuer has a bug or am I correct to assume that is in fact the caBundle? @pearj @lknite do either of you have any ideas regarding the ca.crt or had any issues dealing with it?