Cloud Autoscaling API Errors 100%

[mramteke]

In my GCP console under APIs and Services Section I see Cloud Autoscaling API failing 100% of the time for below API method:

google.cloud.autoscaling.internal.v1alpha1.MetricsReceiver.StreamingUpdateMetricsValues

I am unable to debug the cause of this error. We have HPA configured in our GKE cluster, and I don't see any errors anywhere. Can someone help me debug why this API is failing.

[garisingh]

Could you tell us what GKE version your cluster is running and in what region it's in?

[mramteke]

I have the same issue with my Prod and Dev cluster @garisingh 
Dev Version: 1.26.10-gke.1101000  Zone: us-west2a 
Prod Version: 1.27.11-gke.1062004  Region: us-west2

The latency,median is exactly same in both the clusters: 536,870ms

[mramteke]

Hey @garisingh Do you have any other queries or update.

[garisingh]

The error itself is an odd one and isn't actually indicative of anything.   It's being looking into, but in the meantime, you can create a log exclusion filter to avoid the log messages / costs: