inconsistent(?) metrics: argo_workflows_count

[static-moonlight]

The metric argo_workflows_count, provided by the /metrics endpoint is not consistent with the current state of my Argo Workflows instance and what the user interface shows:

# HELP argo_workflows_count Number of Workflows currently accessible by the controller by status (refreshed every 15s)
# TYPE argo_workflows_count gauge
argo_workflows_count{status="Error"} 1
argo_workflows_count{status="Failed"} 47
argo_workflows_count{status="Pending"} 0
argo_workflows_count{status="Running"} 6
argo_workflows_count{status="Succeeded"} 1059

I would expect that the metrics and the values in the user interface are identical (aside from a 15 sec delay). Technically speaking: that the values come from the same origin.

relevant part of my config:

metricsConfig:
  enabled: true
  path: /metrics
  port: 9090
persistence:
  archive: false
  connectionPool:
    maxIdleConns: 100
    maxOpenConns: 0
    connMaxLifetime: 0s
  nodeStatusOffLoad: true
  postgresql:
    host: database.argo
    port: 5432
    database: argo
    [...]
artifactRepository:
  s3:
    [...]
workflowDefaults:
  spec:
    ttlStrategy:
      [...]

What am I missing?

[Joibel]

Are you looking at all namespaces? the UI is usually filtered.

What do you get from kubectl get -A workflows?

[static-moonlight]

Yes, all namespaces. I removed all filters before I took that screenshot.

What do you get from kubectl get -A workflows

The output is consistent with the ui. It shows me 11 running workflows and 4 failed, across multiple namespaces.

[agilgur5]

Superseded by #13296