X509 certificates with SSO issuer #5079

LeJav · 2021-02-10T21:29:54Z

LeJav
Feb 10, 2021

Hello,

I am using argo workflow with SSO and OpenId Connect (implemented with keycloak).
It is working fine when everything is http.
But when going with https I have a problem with issuer.
server crashes because keycloak certificate authority is not known:

 msg="Get https://xxxxxxxxx/auth/realms/master/.well-known/openid-configuration: x509: certificate signed by unknown authority"

How to solve it?
I have not found any option in order to ignore SSL verification.
Will it work if I set the CA cert in configmap and mount it under /etc/ssl/certs?

Thanks for your advices.

Answered by LeJav

Feb 12, 2021

For those who have the same problem, I have solved it by setting the ca cert in a configmap and mounting it in a file under /etc/ssl/certs

View full answer

LeJav · 2021-02-12T08:26:12Z

LeJav
Feb 12, 2021
Author

For those who have the same problem, I have solved it by setting the ca cert in a configmap and mounting it in a file under /etc/ssl/certs

4 replies

yogendramummaneni Jul 13, 2022

I am also seeing the same issue even after following the below steps:

Added a configmap
Updated the values.yaml with below and tried to install the helm chart and pod is going into CrashLoopBackOff state, logs are below:

  volumeMounts:
     - name: certificate
       mountPath: /etc/ssl/certs/rootCA.crt # your self-signed CA part inside the secret
       #subPath: ca.crt
  volumes:
     - name: certificate
       configMap:
          name: configcert

kubectl logs -f argo-keyclock-argo-workflows-server-66d6b99ffc-n7rn2 -n argo
time="2022-07-13T15:34:32.128Z" level=info msg="config map" name=argo-keyclock-argo-workflows-workflow-controller-configmap
Error: Get "https://keycloak.xxx-xxxx.com/auth/realms/argocd/.well-known/openid-configuration": x509: certificate signed by unknown authority
Usage:
  argo server [flags]

Examples:
See https://argoproj.github.io/argo-workflows/argo-server/

Any idea what mistake I am doing here?

Thanks inadvance.

LeJav Jul 17, 2022
Author

How is your configmap?
On my side I have a key for the crt and I have set subPath.

yogendramummaneni Jul 18, 2022

@LeJav Thanks for the reply!

Now I created a key using this command:

kubectl -n argo create configmap custom-root-cert --from-file=ca-cert=/home/dell/dev-vm/certs/rootCA.crt

and mounting like this inside values.yaml

  volumeMounts:
    - name: certificate
      mountPath: /etc/ssl/certs/rootCA.crt # your self-signed CA part inside the secret
      subPath: ca-cert

  volumes:
    - name: certificate
      configMap:
        name: custom-root-cert

Still unable to mount the certificate under /etc/ssl/certs/rootCA.crt. The file rootCA.crt is being created as empty. I have checked on the overlay file system.

root@kube-cp01:~# cd /var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged
root@kube-cp01:/var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged# ls
bin  dev  etc  home  proc  sys  tmp  var
root@kube-cp01:/var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged# cd etc/ssl/certs/
root@kube-cp01:/var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged/etc/ssl/certs# ls
ca-certificates.crt  rootCA.crt
root@kube-cp01:/var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged/etc/ssl/certs# more rootCA.crt
root@kube-cp01:/var/lib/docker/overlay2/067e664b1eba818203e95bd3a70097ed156899fd7215ec4e666133ca9262d34c/merged/etc/ssl/certs#

Any ideas what could be the wrong here?

LeJav Jul 20, 2022
Author

Hello,
Your configuration seems correct.
It is normal that you do not see it when looking into /var/lib/docker/overlay2/... as it is mounted.
Can you check inside the pod? kubectl exec -it .... -- sh and check if you see your certificate under /etc/ssl/certs

sarabala1979 · 2021-09-10T13:32:19Z

sarabala1979
Sep 10, 2021
Maintainer

Fixed in ae7eeeb I am releasing 3.1.10 today

…

Sent from my iPhone

On Sep 9, 2021, at 11:13 PM, Peter O. ***@***.***> wrote: @LeJav did you do anything else beside mounting your CA pem file under /etc/ssl/certs? That's what I did and I am still getting this error. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

0 replies

peteroruba · 2021-09-11T06:18:36Z

peteroruba
Sep 11, 2021

@sarabala1979 thanks for your response, however, I deleted my question, because it turned out I used the wrong certificate. With the correct one the procedure as explained by @LeJav worked for me as well.

1 reply

chenbr May 2, 2022

did you change the deployment to use the configmap? do you have an example?

LeJav · 2022-05-03T16:09:53Z

LeJav
May 3, 2022
Author

I have currently no access to my configuration. I will describe next week the modifications I have done to the Helm chart in order to mount the cert configmap in the pod.

0 replies

sharadhirao · 2023-02-14T08:07:57Z

sharadhirao
Feb 14, 2023

Hi @LeJav
I have created config map as:
kubectl -n argo create configmap custom-root-cert --from-file=ca-cert=rootCA.crt

and my values.yal for volume looks like this:

volumeMounts:
- name: certificate
  mountPath: /etc/ssl/certs/rootCA.crt # your self-signed CA part inside the secret
  subPath: ca-cert

volumes:
- name: certificate
  configMap:
    name: custom-root-cert

still I am facing the issue of 509 error. Can you help where I am going wrong?

0 replies

xmsanchez · 2023-03-15T17:21:27Z

xmsanchez
Mar 15, 2023

I got the same issue. I got this working in another environment and mounting the certs in /etc/ssl/certs is working there. I don't undertstand why it doesn't work in the new environment.

Error: Get "https://dex.pre.example.com/.well-known/openid-configuration": x509: certificate signed by unknown authority

I double checked the CA certificates, in fact I can spin up a curl container and it works perfectly with the same CAs when querying the SSO endpoint.

Any idea on how to debug this further? Log level in debug mode and enabling --verbose does not help at all.

3 replies

xmsanchez Mar 23, 2023

I fixed it. In the end it was an issue with my CA bundle. I updated it properly and it works.

As suggested in the accepted answer you must mount it at /etc/ssl/certs

Jaydee94 Apr 24, 2023

Hey @xmsanchez,

I have the same issue with my Argo workflows setup. How did you fix your ca bundle?

I've created a configmap with my custom ca in pem format and mounted it into /etc/ssl/certs/ca.crt

After a pod restart I still get the x509 certificate is signed by a unknown authority error.

Any ideas?

Jaydee94 Apr 25, 2023

Never mind. My ca bundle was also broken. The pem format was incorrect.
Works fine now.

LeJav · 2023-03-15T22:59:54Z

LeJav
Mar 15, 2023
Author

I am sorry but I am not working any more with the configuration with the certificates mounted in the pod.
I am really sorry but I cannot test and help...

0 replies

aydinseven7 · 2024-06-25T12:30:05Z

aydinseven7
Jun 25, 2024

Another reminder since I had to debug a bit to get it to work:

add the full chain
mount it to /etc/ssl/certs/, don't add specific cert name

inside values.yaml:

server:
  volumeMounts:
  - name: intermediate-ca
    mountPath: /etc/ssl/certs/
  volumes:
  - name: intermediate-ca
    secret:
      secretName: intermediate-ca

ca-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: intermediate-ca
  namespace: argo
stringData:
  root-ca.crt: |
    CERT1
  intermediate-ca.crt: |
    CERT2
type: Opaque

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

X509 certificates with SSO issuer #5079

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 8 comments 8 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

X509 certificates with SSO issuer #5079

Replies: 8 comments · 8 replies

LeJav Feb 12, 2021 Author

LeJav Jul 17, 2022 Author

LeJav Jul 20, 2022 Author

sarabala1979 Sep 10, 2021 Maintainer

LeJav May 3, 2022 Author

LeJav Mar 15, 2023 Author

Replies: 8 comments 8 replies

LeJav
Feb 12, 2021
Author

LeJav Jul 17, 2022
Author

LeJav Jul 20, 2022
Author

sarabala1979
Sep 10, 2021
Maintainer

LeJav
May 3, 2022
Author

LeJav
Mar 15, 2023
Author