GKE Workload Identity Federation Troubleshooting Skill
Use this skill to systematically diagnose why a Pod using Workload Identity
Federation for GKE cannot authenticate to Google Cloud APIs. Typical symptoms:
HTTP/403 ... Permission 'iam.serviceAccounts.getAccessToken' denied on resourcegoogle.auth.exceptions ... could not find default credentials /
ComputeEngineCredentials cannot find the metadata server- API calls that unexpectedly use the node's default Compute Engine service
account instead of the workload's identity.
Read-only boundary
This skill is diagnostic and non-interactive. It only reads cluster,
IAM, and logging state and proposes fixes as commands or GitOps manifest
changes for a human to apply. It must never create or modify IAM bindings,
KSA annotations, node pools, or clusters automatically. When evidence is missing
or the fix requires a privileged change, summarize findings and hand off to a
human (see Step 7).
Output discipline (apply to every conclusion)
When you report a diagnosis, always:
- Name the single most-likely root cause (not an open-ended list of
possibilities).
- Explicitly rule out the other plausible causes, citing the evidence that
excludes them. In particular, when the cause is node-pool configuration,
state plainly that it is not a KSA annotation or IAM binding problem;
when the cause is a missing role on a target resource, state that Workload
Identity itself is not misconfigured.
- Give the exact remediation as a proposed change for a human — a concrete
gcloud command (or GitOps manifest edit) with the real identifiers filled
in — and never apply it automatically.
Diagnostic Workflow
Step 0: Context discovery & time window
Collect (from the user or the failing resource): PROJECT_ID, PROJECT_NUMBER,
CLUSTER, cluster LOCATION, NAMESPACE, the KSA the Pod runs as, the
node and node pool the Pod is scheduled on (used in Step 2), the target
resource / API being called, and the exact error string. Define a time
window around the first observed failure for log queries.
# Resolve the project number (used in the direct-binding principal identifier).
gcloud projects describe "{PROJECT_ID}" --format="value(projectNumber)"
# Confirm which KSA the workload runs as.
kubectl get pod "{pod_name}" -n "{namespace}" \
-o jsonpath='{.spec.serviceAccountName}'
# Identify the node the Pod runs on, then the node pool that node belongs to
# (Step 2 checks the node pool's Workload Identity mode).
NODE=$(kubectl get pod "{pod_name}" -n "{namespace}" -o jsonpath='{.spec.nodeName}')
kubectl get node "$NODE" \
-o jsonpath='{.metadata.labels.cloud\.google\.com/gke-nodepool}'
Step 1: Capture the exact error signature
Read the workload's own logs and the gke-metadata-server logs to classify the
failure.
kubectl logs "{pod_name}" -n "{namespace}" --all-containers --prefix
kubectl describe pod "{pod_name}" -n "{namespace}"
Equivalent via Cloud Logging (preferred for historical events). Open it as a
Logs Explorer deep link — URL-encode the query and append the project and
time window:
https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}
(encode / as %2F, or use ;duration=PT1H for a rolling hour):
resource.type="k8s_container"
resource.labels.namespace_name="{namespace}"
resource.labels.pod_name="{pod_name}"
severity>=WARNING
Classify the signature:
iam.serviceAccounts.getAccessToken denied / HTTP/403 → the workload is
using the GSA impersonation path. This is expected for the legacy setup,
but if you intend to use direct binding, it means the KSA still carries
a leftover iam.gke.io/gcp-service-account annotation (or the client SDK is
configured to impersonate) and is unintentionally impersonating a GSA; go to
Step 3.403 PERMISSION_DENIED on the target API/resource (no getAccessToken
in the error) → the resolved identity lacks the required IAM role on that
resource; go to Step 4.could not find default credentials / cannot find the metadata server →
metadata-server connectivity or a startup race; go to Step 5.- Calls succeed but as the node default service account → Workload
Identity is not in effect for this node pool; go to Step 2.
Step 2: Verify Workload Identity is enabled (cluster + node pool)
Both the cluster and the node pool the Pod runs on must have Workload
Identity enabled. A node pool with GCE_METADATA (instead of GKE_METADATA)
causes Pods to fall back to the node's default Compute Engine service account.
# Cluster must have a workload identity pool (PROJECT_ID.svc.id.goog).
gcloud container clusters describe "{cluster}" --location "{location}" \
--format="value(workloadIdentityConfig.workloadPool)"
# Node pool must have workloadMetadataConfig.mode = GKE_METADATA.
gcloud container node-pools describe "{node_pool}" --cluster "{cluster}" \
--location "{location}" \
--format="value(config.workloadMetadataConfig.mode)"
- Empty workload pool → Workload Identity is not enabled on the cluster.
- Node-pool mode is
GCE_METADATA (or empty) → the node pool is not using
the GKE metadata server; this is the usual cause of "runs as the node
default service account". Remediation: enable
--workload-metadata=GKE_METADATA on the node pool (propose to a human;
recreates nodes). This is a node-pool configuration problem — not a KSA
annotation or IAM binding problem — so do not change KSA annotations or IAM
bindings to fix it. When the symptom is "runs as the node default service
account", say so explicitly: the root cause is the node pool's
workloadMetadataConfig.mode, and the KSA annotation and IAM bindings are
ruled out as the cause. Propose the exact fix, e.g.:
gcloud container node-pools update "{node_pool}" --cluster "{cluster}" \
--location "{location}" --workload-metadata=GKE_METADATA
Step 3: Verify the KSA → identity binding
There are two supported models. Prefer direct binding (current default);
treat GSA impersonation as the legacy path.
How the two models fail differently: with direct binding the KSA
principal accesses resources directly, so failures show up as a plain 403 PERMISSION_DENIED on the target API (fix in Step 4). A 403 iam.serviceAccounts.getAccessToken instead means an impersonation attempt
— intended under the legacy path, or unintended if a leftover
iam.gke.io/gcp-service-account annotation remains on a KSA that was meant to
use direct binding.
(a) Direct KSA binding (no GSA impersonation). The KSA principal is granted
roles directly. Construct the principal identifier and search for its bindings:
principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}
(b) Legacy: KSA + GSA impersonation. The KSA must be annotated to point at a
GSA, and the KSA must hold roles/iam.workloadIdentityUser on that GSA.
# The KSA annotation must reference the intended GSA.
kubectl get serviceaccount "{ksa_name}" -n "{namespace}" \
-o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}'
# The GSA's IAM policy must bind the KSA member to workloadIdentityUser.
gcloud iam service-accounts get-iam-policy \
"{gsa_name}@{project_id}.iam.gserviceaccount.com" \
--format=json
# Expect a binding: role roles/iam.workloadIdentityUser,
# member serviceAccount:{PROJECT_ID}.svc.id.goog[{NAMESPACE}/{KSA_NAME}]
If the annotation is present but the binding is missing, the binding was
likely removed — check the setIamPolicy audit logs around the failure time
to find the responsible principal.
Step 4: Verify IAM permissions on the target resource
Even with a correct binding, the identity (the KSA principal for direct binding,
or the GSA for legacy) must hold the role required by the API call (for example
roles/storage.objectViewer). The standard IAM Policy Troubleshooter has
limited support for Workload Identity principals; use Cloud Asset Inventory to
search all IAM policies for the principal instead.
# Direct binding: search for the KSA principal's bindings across the project.
gcloud asset search-all-iam-policies \
--scope="projects/{PROJECT_ID}" \
--query='policy:"{PROJECT_ID}.svc.id.goog"'
# Legacy: search for the GSA's bindings on the target resource's project.
gcloud asset search-all-iam-policies \
--scope="projects/{TARGET_PROJECT_ID}" \
--query='policy:"{gsa_name}@{project_id}.iam.gserviceaccount.com"'
If no binding grants the required role on the target resource, that missing
role is the root cause (and Workload Identity itself is not
misconfigured). Present the fix as a proposed change for a human to apply —
the exact add-iam-policy-binding with the real principal and role — never
applying it automatically. For example, for direct binding on a project-level
resource:
gcloud projects add-iam-policy-binding "{TARGET_PROJECT_ID}" \
--member="principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}" \
--role="{REQUIRED_ROLE}" # e.g. roles/storage.objectViewer
Step 5: GKE metadata server & connectivity
Applies when the signature is could not find the metadata server, cannot find the metadata server, or a connection/timeout error. The gke-metadata-server
DaemonSet (in kube-system) brokers the token exchange on each node; requests
to 169.254.169.254 are redirected to it, so a Pod fails closed if it cannot
reach a healthy metadata-server Pod on its node.
(a) Check gke-metadata-server Pod health on the workload's node.
# The DaemonSet Pods must be healthy on the workload's node.
kubectl get pods -n kube-system -l k8s-app=gke-metadata-server -o wide
# A gke-metadata-server Pod can be OOM-evicted when the cluster has many
# (>3,000) Kubernetes service accounts. Look for CrashLoopBackOff, then confirm
# the eviction was OOMKilled.
kubectl get pods -n kube-system | grep CrashLoopBackOff
kubectl describe pod {gke_metadata_server_pod} --namespace=kube-system | grep OOMKilled
(b) Inspect the gke-metadata-server logs (historical, via Cloud Logging;
open as a Logs Explorer deep link, see Step 1):
resource.type="k8s_container"
resource.labels.namespace_name="kube-system"
labels."k8s-pod/k8s-app"="gke-metadata-server"
severity>=WARNING
(c) Connectivity test from the affected Pod — it must reach the metadata
server and (for some client libraries) resolve its DNS name:
# Token endpoint via the hardcoded metadata IP (a healthy path returns a token).
kubectl exec {pod_name} -n {namespace} -- \
curl -sS -H 'Metadata-Flavor: Google' \
'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token'
Possible causes:
- Startup race: the metadata server needs a few seconds after a Pod
starts. Applications that authenticate immediately may fail; the fix is
application-side retries or an
initContainer that waits for the metadata
server, not a cluster change.
- DNS: some client libraries resolve
metadata.google.internal; broken
in-cluster DNS surfaces as cannot find the metadata server. As a
workaround, set GCE_METADATA_HOST=169.254.169.254 to skip DNS resolution.
- Network policy / egress: a cluster network policy must allow egress to
169.254.169.254/32 on port 80 (GKE Dataplane V2), or
169.254.169.252/32 on port 988. A default-deny egress rule that blocks
HTTPS to the public Security Token Service (sts.googleapis.com:443)
produces 504 Gateway Timeout or context deadline exceeded.
gke-metadata-server crashing / high restarts: see (a) — reduce the
number of Kubernetes service accounts (<3,000) to restore functionality.
Step 6: "Invalid form of account ID" edge case (legacy GSA impersonation)
Applies only to the legacy GSA impersonation setup (a KSA annotated with
iam.gke.io/gcp-service-account), not to direct principal binding.
Signature: an operation that requires an IAM service account email (for
example, manually creating a Cloud Storage signed URL) fails with:
ERROR: Invalid form of account ID SERVICEACCOUNT_NAME.svc.id.goog.
Should be [Gaia ID | Email | Unique ID | ] of the account
Cause: for a KSA linked to a GSA via annotation, the GKE metadata server by
default returns SERVICEACCOUNT_NAME.svc.id.goog as the identifier, which is
not a valid IAM service account email. This is not a missing IAM binding and
does not require reverting the setup.
Fix (propose to a human): add the
iam.gke.io/return-principal-id-as-email="true" annotation to the Pod's KSA so
the metadata server returns the identity in the expected form:
kubectl annotate serviceaccount "{ksa_name}" --namespace "{namespace}" \
iam.gke.io/return-principal-id-as-email="true"
Step 7: Remediation boundary & escalation
- Present the root cause + evidence (the exact error, the missing binding
/ annotation / node-pool mode, or the metadata-server state). Include a
Cloud Logging deep link (see Step 1) to the supporting entries.
- Propose the fix as a command or GitOps manifest change for a human to
apply — never modify IAM, KSAs, or node pools automatically.
Escalate (instead of proposing more self-service diagnostics) when either:
- the relevant logs are unavailable (excluded by a filter, or past the log
bucket's retention); or
- the identity/binding is correct, the target-resource IAM is correct, and the
metadata server is healthy, yet the failure persists (undetermined root
cause).
In those cases: state the limitation plainly, summarize the findings gathered
(cluster/node-pool config, bindings, annotations, metadata-server state, and any
setIamPolicy audit logs), and route to GKE support / engineering escalation.
Do not fabricate a diagnosis when evidence is missing.
References
This skill is derived from public Google Cloud documentation: