Github sso with dex not working

[ghost]

Summary

Hi,

Version 3.3.4 (namespaced) installed from yaml

I have trouble setting up github sso for github workflows. In browser i can see 401 on /api/v1/info and i am redirected back to login screen with Failed to load version/info Error: Unsuccessful HTTP response. In argo-server log:

time="2022-05-04T09:10:05.827Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2022-05-04T09:10:05Z" grpc.time_ms=0.048 span.kind=server system=grpc
time="2022-05-04T09:10:17.149Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2022-05-04T09:10:17Z" grpc.time_ms=0.719 span.kind=server system=grpc
time="2022-05-04T09:10:17.151Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=ListCronWorkflows grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2022-05-04T09:10:17Z" grpc.time_ms=0.032 span.kind=server system=grpc
time="2022-05-04T09:10:17.154Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2022-05-04T09:10:17Z" grpc.time_ms=0.043 span.kind=server system=grpc
time="2022-05-04T09:10:17.851Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2022-05-04T09:10:17Z" grpc.time_ms=0.043 span.kind=server system=grpc
time="2022-05-04T09:10:35.023Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2022-05-04T09:10:35Z" grpc.time_ms=0.044 span.kind=server system=grpc
time="2022-05-04T09:10:35.032Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2022-05-04T09:10:35Z" grpc.time_ms=0.038 span.kind=server system=grpc
time="2022-05-04T09:10:35.032Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=ListCronWorkflows grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2022-05-04T09:10:35Z" grpc.time_ms=0.032 span.kind=server system=grpc
time="2022-05-04T09:10:35.035Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2022-05-04T09:10:35Z" grpc.time_ms=0.032 span.kind=server system=grpc
time="2022-05-04T09:10:35.373Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2022-05-04T09:10:35Z" grpc.time_ms=0.045 span.kind=server system=grpc

Dex logs looks as follow:

time="2022-05-04T10:39:48Z" level=info msg="login successful: connector \"github\", username=\"....\", preferred_username=\"....\", email=\"....\", groups=[\"..." \"....\" \"...." \"....\" \"....\"]"

I have my dex configured like this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
  name: dex
data:
  admin.enabled: "false"
  users.anonymous.enabled: "false"
  users.session.duration: "8h"
  url: https://argo-workflows.mydomain.com
  dex.config: |
    issuer: https://argo-workflows.mydomain.com/api/dex
    logger:
      level: "debug"
    storage:
      type: memory
      # config:
      #   inCluster: true
    web:
      http: 0.0.0.0:5556
    grpc:
      addr: 0.0.0.0:5557
    telemetry:
      http: 0.0.0.0:5558
    oauth2:
      skipApprovalScreen: true
    connectors:
    - type: github
      id: github
      name: GitHub
      config:
        clientID: github oauth client id
        clientSecret: github oauth client secret
        redirectURI: https://argo-workflows.mydomain.com/api/dex/callback
        orgs:
        - name: myorg
    staticClients:
#id matches sso client id in workflow-controller configmap, i dont get any errors
    - idEnv: ARGO_WORKFLOWS_SSO_CLIENT_ID
      name: Argo Workflows
      redirectURIs:
        - https://argo-workflows.mydomain.com/auth/callback
      secretEnv: ARGO_WORKFLOWS_SSO_CLIENT_SECRET

workflow-controller configmap:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:
....
  sso: |
    issuer: http://dex:5556/api/dex
    issuerAlias: https://argo-workflows.mydomain.com/api/dex
    clientId:
      name: argo-workflows-dex-creds
      key: clientName
    clientSecret:
      name: argo-workflows-dex-creds
      key: clientSecret
    redirectUrl: https://argo-workflows.mydomain.com/auth/callback
    scopes:
      - groups
    rbac:
      enabled: true

My deployments are exposed via ALB with dedicated conditions to cover dex and grpc:

    alb.ingress.kubernetes.io/conditions.argo-server-grpc: |
      [{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "Content-Type", "values":["application/grpc"]}}]
    alb.ingress.kubernetes.io/conditions.dex: |
      [{"field": "path-pattern", "PathPatternConfig": {"Values": ["/api/dex", "/api/dex/*"]}}]

argo-server started with following parameters:

      - args:
        - server
        - --namespaced
        # engicustom
        - --auth-mode
        - client
        - --auth-mode
        - sso
        - --loglevel
        - debug

Do you have any suggestions what am i doing wrong?
Kind Regards