How to resolve Istio 503 NC cluster_not_found on Kubernetes

The Istio 503 NC cluster_not_found error typically occurs when the service mesh cannot find a destination cluster for routing. This error is usually seen in scenarios involving Istio's Envoy proxy and can be caused by several issues, such as incorrect configuration, missing service discovery, or routing rules misconfiguration.

Here’s how you can troubleshoot and resolve the error:

1. Check Service Discovery

Verify the service exists: Ensure the destination service is running in the cluster and is discoverable by Istio.

Inspect Istio's service registry:


istioctl proxy-config endpoints <pod-name>.<namespace>

Replace <pod-name> and <namespace> with the name of the pod and its namespace. Ensure the destination service is listed.

Istio 503 nc cluster_not_found example
(created by author using Bing Image Creator)

2. Validate Virtual Service and Destination Rule

Confirm that the VirtualService and DestinationRule are properly defined for the service.
Verify that the host field matches the actual service name (including the namespace if necessary):


apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example
spec:
  hosts:
    - service-name.namespace.svc.cluster.local
  http:
    - route:
        - destination:
            host: service-name.namespace.svc.cluster.local
            port:
              number: 8080

3. Check Sidecar Injection

Ensure that sidecar injection is enabled for the namespaces where the source and destination services are running:


kubectl get namespace -L istio-injection

If not enabled, annotate the namespace:



kubectl label namespace <namespace> istio-injection=enabled --overwrite

Confirm the sidecar proxy is running:


kubectl get pods -n <namespace> -o jsonpath='{.items[*].spec.containers[*].name}' | grep istio-proxy

4. Verify Connectivity and Endpoints

Check if the endpoints of the destination service are correctly registered:


kubectl get endpoints -n <namespace>

Confirm that the pods backing the service are healthy and ready:


kubectl get pods -n <namespace>

5. Inspect Gateway and Ingress Configuration

If the error involves traffic coming through an Istio Gateway, ensure the Gateway is properly configured:


apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: example-gateway
spec:
  selector:
    istio: ingressgateway # Use Istio ingress gateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

Verify the Gateway and VirtualService are linked correctly.

6. Check DNS Resolution

Ensure the DNS names of the services resolve correctly inside the cluster:


kubectl exec <pod-name> -n <namespace> -- nslookup <service-name>

7. Examine Logs

Inspect logs from the sidecar proxy for the source pod:


kubectl logs <pod-name> -n <namespace> -c istio-proxy

Check the Istio control plane logs:


kubectl logs -n istio-system -l app=istiod

8. Validate Istio Configuration Consistency

Ensure Istio's configuration is consistent across namespaces:
```
istioctl analyze
```

9. Sync or Restart Pods

Restart the affected pods to re-trigger service discovery:


kubectl rollout restart deployment <deployment-name> -n <namespace>

10. Debug Using Envoy Config

Inspect the Envoy configuration of the sidecar proxy:


istioctl proxy-config clusters <pod-name>.<namespace>

Look for the expected cluster and its configuration.

Feel free to use comment section for further query on resolving 503 NC cluster not found error on Kubernetes

Tag:

"Troubleshooting Istio 503 Errors: Resolving NC cluster_not_found Effectively"

"A Step-by-Step Guide to Fixing Istio 503 NC cluster_not_found Errors"

"Understanding and Resolving the Istio 503 NC cluster_not_found Error"

"Istio Service Mesh Debugging: Fixing 503 NC cluster_not_found"

"Comprehensive Troubleshooting for Istio 503 Errors in Kubernetes"

"Istio 503 Errors Demystified: How to Resolve NC cluster_not_found Issues"

"Debugging Istio Networking Errors: A Guide to 503 NC cluster_not_found"

"Istio retry on 503"

"Istio 503 nc cluster_not_found kubernetes"

"Istio 503 nc cluster_not_found example"

"GET HTTP/1.1'' 503 NC cluster_not_found"

"Istio 503 no healthy upstream"

"Cluster_not_found envoy"