Service Mesh

Implement service-to-service communication management with mTLS, traffic shaping, observability, and policy enforcement using Istio or Linkerd.

When to Use

• Securing microservice communication with automatic mTLS.
• Implementing canary deployments, traffic splitting, or A/B testing.
• Adding circuit breakers, retries, and timeouts without changing application code.
• Gaining service-level observability (latency, error rates, request volume).
• Enforcing authorization policies between services.

Prerequisites

• Kubernetes cluster (1.26+) with kubectl configured.
• Helm 3 installed (for some installation methods).
• Sufficient cluster resources (Istio control plane needs ~2 GB RAM).
• For Istio: istioctl CLI installed.
• For Linkerd: linkerd CLI installed.

Istio Installation

Install with istioctl

# Download istioctl
curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH

# Install with the production profile
istioctl install --set profile=default -y

# Or use the demo profile (includes all addons, good for learning)
istioctl install --set profile=demo -y

# Verify installation
istioctl verify-install

# Check running components
kubectl get pods -n istio-system

Enable Sidecar Injection

# Enable automatic sidecar injection for a namespace
kubectl label namespace default istio-injection=enabled

# Verify label
kubectl get namespace default --show-labels

# Restart existing pods to inject sidecars
kubectl rollout restart deployment -n default

# Check sidecar status
kubectl get pods -n default -o jsonpath='{range .items[*]}{.metadata.name}{" containers: "}{range .spec.containers[*]}{.name}{" "}{end}{"\n"}{end}'

Install Observability Addons

# Install Kiali, Prometheus, Grafana, Jaeger
kubectl apply -f samples/addons/prometheus.yaml
kubectl apply -f samples/addons/grafana.yaml
kubectl apply -f samples/addons/jaeger.yaml
kubectl apply -f samples/addons/kiali.yaml

# Wait for rollout
kubectl rollout status deployment/kiali -n istio-system

# Access dashboards
istioctl dashboard kiali
istioctl dashboard grafana
istioctl dashboard jaeger

Traffic Management

VirtualService (Routing Rules)

# virtualservice.yaml — canary deployment with traffic split
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app
  namespace: default
spec:
  hosts:
    - my-app
  http:
    # Header-based routing (canary testers)
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: my-app
            subset: canary
    # Percentage-based traffic split
    - route:
        - destination:
            host: my-app
            subset: stable
          weight: 90
        - destination:
            host: my-app
            subset: canary
          weight: 10
      timeout: 30s
      retries:
        attempts: 3
        perTryTimeout: 10s
        retryOn: gateway-error,connect-failure,refused-stream

DestinationRule (Subsets and Connection Policy)

# destinationrule.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-app
  namespace: default
spec:
  host: my-app
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
  subsets:
    - name: stable
      labels:
        version: v1
    - name: canary
      labels:
        version: v2

Gateway (Ingress Traffic)

# gateway.yaml — expose service to external traffic
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: app-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: app-tls-cert  # Kubernetes secret
      hosts:
        - app.example.com
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - app.example.com
      tls:
        httpsRedirect: true
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: app-external
  namespace: default
spec:
  hosts:
    - app.example.com
  gateways:
    - app-gateway
  http:
    - route:
        - destination:
            host: my-app
            port:
              number: 8080

mTLS Configuration

Strict mTLS (Cluster-Wide)

# peer-authentication.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system  # Applies to entire mesh
spec:
  mtls:
    mode: STRICT

Permissive mTLS (Per Namespace)

# Allow both plaintext and mTLS during migration
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: legacy-apps
spec:
  mtls:
    mode: PERMISSIVE

Verify mTLS Status

# Check mTLS status for a namespace
istioctl x describe pod <pod-name> -n default

# View TLS configuration
istioctl proxy-config cluster <pod-name>.default --fqdn my-app.default.svc.cluster.local -o json | grep -A5 "tlsContext"

# Verify with istioctl authn
istioctl authn tls-check <pod-name>.default my-app.default.svc.cluster.local

Authorization Policies

# authz-policy.yaml — only allow frontend to call API
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: api-access
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-api
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/default/sa/frontend"
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/api/*"]
---
# Deny all other traffic to api
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-api
  action: DENY
  rules:
    - from:
        - source:
            notPrincipals:
              - "cluster.local/ns/default/sa/frontend"

Circuit Breaking

# circuit-breaker.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-api-circuit-breaker
spec:
  host: my-api
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 50
      http:
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
        maxRetries: 3
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 15s
      baseEjectionTime: 60s
      maxEjectionPercent: 100

Linkerd Installation

# Install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh
export PATH=$HOME/.linkerd2/bin:$PATH

# Validate cluster prerequisites
linkerd check --pre

# Install Linkerd CRDs
linkerd install --crds | kubectl apply -f -

# Install Linkerd control plane
linkerd install | kubectl apply -f -

# Verify installation
linkerd check

# Inject sidecar into a namespace
kubectl get deploy -n my-app -o yaml | linkerd inject - | kubectl apply -f -

# Or annotate namespace for auto-injection
kubectl annotate namespace my-app linkerd.io/inject=enabled

# View live traffic dashboard
linkerd viz install | kubectl apply -f -
linkerd viz dashboard

Linkerd Traffic Split (SMI)

# traffic-split.yaml
apiVersion: split.smi-spec.io/v1alpha4
kind: TrafficSplit
metadata:
  name: my-app-split
  namespace: default
spec:
  service: my-app
  backends:
    - service: my-app-stable
      weight: 900
    - service: my-app-canary
      weight: 100

Debugging

# Istio: check proxy configuration
istioctl proxy-config routes <pod-name>.default
istioctl proxy-config clusters <pod-name>.default
istioctl proxy-config listeners <pod-name>.default

# Istio: analyze configuration for issues
istioctl analyze -n default

# Istio: proxy debug logs
istioctl proxy-config log <pod-name>.default --level debug

# Linkerd: check proxy stats
linkerd viz stat deploy -n default
linkerd viz top deploy/my-app -n default
linkerd viz edges deploy -n default

Troubleshooting

Symptom	Cause	Fix
Sidecar not injected	Missing namespace label	Add istio-injection=enabled label; restart pods
503 errors between services	mTLS mismatch (one side plaintext)	Set PeerAuthentication to PERMISSIVE during migration
High latency after mesh install	Sidecar resource limits too low	Increase sidecar CPU/memory limits in mesh config
VirtualService not routing	Missing DestinationRule subsets	Create matching DestinationRule with subset labels
upstream connect error	Circuit breaker tripped	Check outlier detection settings; increase thresholds
Authorization policy blocks everything	Default deny without matching allow rule	Add explicit ALLOW rule before DENY-all
Kiali shows "Unknown" traffic	Missing sidecar on calling service	Inject sidecar into all communicating services

Related Skills

• load-balancing - Layer 4/7 load balancing outside Kubernetes
• reverse-proxy - Ingress-level proxying
• ai-inference-service-mesh - Mesh patterns for ML workloads
• dns-management - DNS for mesh ingress gateways