Add prefix to workflow pod names for filtering metrics

[ecurbelo]

Summary

Can we add a prefix to the pods names launched by the Workflows?

We tried to set some extra configuration on a custom workflow-controller-configmap , trying for example to use podSpecPatch in workflowDefaults , but with unsuccessful results.

Motivation

The workflow pods finish in a Failed state in different situations, e.g. an error on the WorkflowTemplate manifest. We want to exclude these pods from the KubePodNotReady Prometheus Alert that we have in place. So, our first approach is adding a prefix on the pods names.

Our second approach could be maybe launching these pods in another namespace, but this is a "migration" task that we would like to avoid.

Also, we would want to avoid passing labels to the prometheus metrics to control our cardinality.

Extra notes

We also added the Role and RoleBinding needed to patch the pods:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo-patch-role
  namespace: argo
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argo-patch-rolebinding
  namespace: argo
subjects:
- kind: ServiceAccount
  name: default
  namespace: argo
roleRef:
  kind: Role
  name: argo-patch-role
  apiGroup: rbac.authorization.k8s.io

kube-prometheus-stack KubePodNotReady expression:

sum by (namespace, pod, cluster) (max by (namespace, pod, cluster) (kube_pod_status_phase{job="kube-state-metrics",namespace=~".*",phase=~"Pending|Unknown|Failed"}) * on (namespace, pod, cluster) group_left (owner_kind) topk by (namespace, pod, cluster) (1, max by (namespace, pod, owner_kind, cluster) (kube_pod_owner{owner_kind!="Job"}))) > 0

Thanks!

[agilgur5]

You can use generateName to set individual prefixes for every Workflow and WorkflowTemplate.

But that doesn't really seem like a correct or optimal approach to your problem. Labels are used for filtering in k8s, so I would suggest using those. You could also use some of the existing labels that the Controller automatically adds.

Also, we would want to avoid passing labels to the prometheus metrics to control our cardinality.

I see you added this later. You could limit which labels you retain.

[jswxstw]

Also, we would want to avoid passing labels to the prometheus metrics to control our cardinality.

I don't think passing the prefix of pod names into prometheus metric label will increase the cardinality.

Shouldn't the following two usages have similar effects?

• pod_name label (with prefix)
• pod_name label (without prefix) + pod_name_prefix label

[ecurbelo]

@agilgur5
I tried to update the generateName, name, etc, with a something like like podSpecPatch: '{"metadata": {"generateName": "pipeline-}}' but the pods names were never changed.

Anyways, @agilgur5, @jswxstw, thank you for your answers. You convinced me to give a chance to the labels approach. I will try to enable kube_pod_labels from kube-state-metrics and measure the performance. Right now the workflow pods already have the workflows.argoproj.io/workflow=XXX label, i will start with this. :)