Istio Observability with Jaeger & Kiali Tutorial

Configure comprehensive observability for your Kubernetes service mesh with Jaeger distributed tracing, Kiali visualization, and Prometheus metrics integration. Get complete visibility into microservice communication patterns, performance bottlenecks, and service dependencies.

Prerequisites

Kubernetes cluster with 8GB+ RAM
kubectl with admin access
Helm 3 installed
Basic Kubernetes knowledge

What this solves

Kubernetes service mesh deployments create complex communication patterns between microservices that are difficult to monitor and debug without proper observability tools. This tutorial implements a complete observability stack with Istio service mesh, Jaeger for distributed tracing, Kiali for service mesh visualization, and integrated Prometheus metrics collection.

You'll get real-time visibility into service-to-service communication, performance bottlenecks, error rates, and dependency graphs across your entire mesh topology.

Prerequisites

Running Kubernetes cluster with at least 8GB RAM and 4 CPU cores
kubectl configured with cluster admin access
Helm 3 installed on your system
Basic understanding of Kubernetes concepts and YAML manifests

Step-by-step installation

Install Istio with observability components

Download and install Istio with the demo profile that includes all observability add-ons by default.

curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH
istioctl install --set values.defaultRevision=default --set values.pilot.traceSampling=100.0 -y

Enable automatic sidecar injection

Configure the default namespace to automatically inject Istio sidecar proxies into new pods.

kubectl label namespace default istio-injection=enabled
kubectl get namespace default --show-labels

Install observability add-ons

Deploy Jaeger, Kiali, Prometheus, and Grafana components using Istio's sample configurations.

kubectl apply -f samples/addons/jaeger.yaml
kubectl apply -f samples/addons/kiali.yaml
kubectl apply -f samples/addons/prometheus.yaml
kubectl apply -f samples/addons/grafana.yaml

Configure Jaeger with Elasticsearch backend

Replace the default in-memory Jaeger storage with Elasticsearch for production-grade persistence and scalability.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: istio-system
  labels:
    app: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
      - name: jaeger
        image: jaegertracing/all-in-one:1.51
        env:
        - name: SPAN_STORAGE_TYPE
          value: "elasticsearch"
        - name: ES_SERVER_URLS
          value: "http://elasticsearch.istio-system.svc.cluster.local:9200"
        - name: ES_NUM_SHARDS
          value: "1"
        - name: ES_NUM_REPLICAS
          value: "0"
        ports:
        - containerPort: 16686
          protocol: TCP
        - containerPort: 14250
          protocol: TCP
        - containerPort: 14268
          protocol: TCP

Deploy Elasticsearch for Jaeger storage

Create a dedicated Elasticsearch instance for storing trace data with appropriate resource limits.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: istio-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
        env:
        - name: discovery.type
          value: single-node
        - name: ES_JAVA_OPTS
          value: "-Xms512m -Xmx512m"
        - name: xpack.security.enabled
          value: "false"
        ports:
        - containerPort: 9200
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1"
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: istio-system
spec:
  selector:
    app: elasticsearch
  ports:
  - port: 9200
    targetPort: 9200

kubectl apply -f elasticsearch.yaml

Configure Istio telemetry settings

Enable comprehensive telemetry collection including access logs and custom metrics for better observability.

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: default
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus
  - overrides:
    - match:
        metric: ALL_METRICS
      tagOverrides:
        destination_service_name:
          operation: UPSERT
          value: "destination_service_name | 'unknown'"
  accessLogging:
  - providers:
    - name: otel
  tracing:
  - providers:
    - name: jaeger

kubectl apply -f telemetry-config.yaml

Configure Kiali with authentication

Set up Kiali dashboard with proper authentication and configure access to the service mesh topology.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali
  namespace: istio-system
data:
  config.yaml: |
    auth:
      strategy: anonymous
    deployment:
      accessible_namespaces:
      - "**"
      ingress_enabled: false
    external_services:
      prometheus:
        url: "http://prometheus.istio-system:9090"
      grafana:
        url: "http://grafana.istio-system:3000"
      tracing:
        enabled: true
        in_cluster_url: "http://jaeger.istio-system:16686/jaeger"
        url: "http://jaeger.istio-system:16686/jaeger"
        use_grpc: false
    server:
      web_root: "/kiali"

kubectl apply -f kiali-config.yaml
kubectl rollout restart deployment/kiali -n istio-system

Deploy sample application for testing

Install the Bookinfo sample application to generate traffic and test the observability stack.

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Configure Prometheus service discovery

Ensure Prometheus can discover and scrape metrics from all Istio components and application pods.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus
  namespace: istio-system
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    rule_files:
    - "/etc/config/recording_rules.yml"
    - "/etc/config/alerting_rules.yml"
    scrape_configs:
    - job_name: 'istioproxy'
      kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
          - istio-system
          - default
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: istio-proxy;http-monitoring
    - job_name: 'istio-mesh'
      kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
          - istio-system
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: istio-telemetry;prometheus
    - job_name: 'pilot'
      kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
          - istio-system
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: istiod;http-monitoring

kubectl apply -f prometheus-config.yaml

Set up port forwarding for dashboard access

Create secure tunnels to access the observability dashboards from your local machine.

# Kiali dashboard
kubectl port-forward svc/kiali 20001:20001 -n istio-system &

Jaeger UI
kubectl port-forward svc/jaeger 16686:16686 -n istio-system &

Grafana dashboard
kubectl port-forward svc/grafana 3000:3000 -n istio-system &

Prometheus UI
kubectl port-forward svc/prometheus 9090:9090 -n istio-system &

Generate traffic for observability testing

Create consistent traffic to the sample application to populate the observability dashboards with meaningful data.

# Get the ingress gateway external IP
export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT

Generate traffic
for i in $(seq 1 100); do
  curl -s "http://$GATEWAY_URL/productpage" > /dev/null
  echo "Request $i completed"
  sleep 1
done

Configure advanced observability features

Set up custom Grafana dashboards

Import Istio-specific Grafana dashboards for comprehensive service mesh monitoring. You can build on the foundation established in our advanced Grafana dashboards tutorial.

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-dashboards
  namespace: istio-system
data:
  istio-mesh.json: |
    {
      "dashboard": {
        "id": null,
        "title": "Istio Mesh Dashboard",
        "tags": ["istio"],
        "panels": [
          {
            "title": "Global Request Volume",
            "type": "graph",
            "targets": [
              {
                "expr": "sum(rate(istio_requests_total[5m]))",
                "legendFormat": "Global Request Rate"
              }
            ]
          },
          {
            "title": "Global Success Rate",
            "type": "singlestat",
            "targets": [
              {
                "expr": "sum(rate(istio_requests_total{response_code!~\"5.*\"}[5m])) / sum(rate(istio_requests_total[5m]))",
                "legendFormat": "Success Rate"
              }
            ]
          }
        ]
      }
    }

kubectl apply -f grafana-dashboards.yaml

Configure alerting rules

Set up Prometheus alerting rules for service mesh health monitoring and automated incident response.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-alerting-rules
  namespace: istio-system
data:
  alerting_rules.yml: |
    groups:
    - name: istio.rules
      rules:
      - alert: IstioHighRequestLatency
        expr: histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket[5m])) by (destination_service_name, le)) > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High request latency on {{ $labels.destination_service_name }}"
          description: "99th percentile latency is {{ $value }}ms for service {{ $labels.destination_service_name }}"
      - alert: IstioHighErrorRate
        expr: sum(rate(istio_requests_total{response_code=~"5.*"}[5m])) / sum(rate(istio_requests_total[5m])) > 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }} for the last 5 minutes"
      - alert: IstioServiceDown
        expr: up{job="istio-mesh"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Istio service is down"
          description: "Istio service {{ $labels.instance }} has been down for more than 5 minutes"

kubectl apply -f alerting-rules.yaml

Verify your setup

Check that all observability components are running and accessible.

# Verify all pods are running
kubectl get pods -n istio-system

Check Istio sidecar injection
kubectl get pods -l app=productpage -o jsonpath='{.items[0].spec.containers[*].name}'

Test Kiali access
curl -s http://localhost:20001/kiali/api/status | jq '.status'

Test Jaeger access
curl -s http://localhost:16686/api/services | jq '.data[]'

Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health == "up") | .labels.job' | sort -u

Dashboard URLs: Once port forwarding is active, access Kiali at http://localhost:20001/kiali, Jaeger at http://localhost:16686, Grafana at http://localhost:3000, and Prometheus at http://localhost:9090.

Configure production security

Enable mTLS for observability components

Secure communication between observability components with mutual TLS authentication.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: observability-mtls
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: observability-access
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: kiali
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/istio-system/sa/kiali-service-account"]
  - to:
    - operation:
        methods: ["GET", "POST"]

kubectl apply -f mtls-policy.yaml

Configure network policies

Implement network-level security policies to restrict traffic between observability components. This builds on concepts from our Kubernetes network policies tutorial.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: observability-network-policy
  namespace: istio-system
spec:
  podSelector:
    matchLabels:
      app: kiali
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: istio-system
    ports:
    - protocol: TCP
      port: 20001
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: istio-system
    ports:
    - protocol: TCP
      port: 9090  # Prometheus
    - protocol: TCP
      port: 16686 # Jaeger

kubectl apply -f network-policy.yaml

Common issues

Symptom	Cause	Fix
No traces in Jaeger	Sidecar injection not enabled	`kubectl label namespace default istio-injection=enabled`
Kiali shows no data	Prometheus not scraping metrics	Check Prometheus targets at `localhost:9090/targets`
High memory usage	Default trace sampling at 100%	Reduce sampling rate: `istioctl install --set values.pilot.traceSampling=1.0`
Elasticsearch connection failed	Service not running or misconfigured	`kubectl logs deployment/elasticsearch -n istio-system`
Port forward connection refused	Service not ready	`kubectl get pods -n istio-system` and wait for Running status
Missing service topology	Insufficient traffic generation	Generate more requests to populate service graph

Next steps

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: capacity planning, failover drills, cost control, and on-call. See how we run infrastructure like this for European teams.

Automated install script

Run this to automate the entire setup

install.sh

#!/usr/bin/env bash
set -euo pipefail

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# Configuration
ISTIO_NAMESPACE="istio-system"
DEFAULT_NAMESPACE="default"
JAEGER_VERSION="1.51"
ELASTICSEARCH_VERSION="8.11.0"

usage() {
    echo "Usage: $0 [OPTIONS]"
    echo "Install Istio observability stack with Jaeger tracing and Kiali dashboard"
    echo ""
    echo "Options:"
    echo "  -h, --help          Show this help message"
    echo "  --cluster-name      Kubernetes cluster name (optional)"
    exit 1
}

log() {
    echo -e "${GREEN}[INFO]${NC} $1"
}

warn() {
    echo -e "${YELLOW}[WARN]${NC} $1"
}

error() {
    echo -e "${RED}[ERROR]${NC} $1"
    exit 1
}

cleanup() {
    warn "Installation failed. Cleaning up..."
    kubectl delete namespace $ISTIO_NAMESPACE --ignore-not-found=true
    exit 1
}

trap cleanup ERR

# Parse arguments
CLUSTER_NAME=""
while [[ $# -gt 0 ]]; do
    case $1 in
        -h|--help)
            usage
            ;;
        --cluster-name)
            CLUSTER_NAME="$2"
            shift 2
            ;;
        *)
            error "Unknown option: $1"
            ;;
    esac
done

# Check if running as root or with sudo
if [[ $EUID -eq 0 ]] && [[ -z "${SUDO_USER:-}" ]]; then
    error "Don't run this script as root. Use sudo instead."
fi

# Auto-detect distribution
if [ -f /etc/os-release ]; then
    . /etc/os-release
    case "$ID" in
        ubuntu|debian) 
            PKG_MGR="apt"
            PKG_INSTALL="apt install -y"
            PKG_UPDATE="apt update"
            ;;
        almalinux|rocky|centos|rhel|ol|fedora) 
            PKG_MGR="dnf"
            PKG_INSTALL="dnf install -y"
            PKG_UPDATE="dnf update -y"
            ;;
        amzn) 
            PKG_MGR="yum"
            PKG_INSTALL="yum install -y"
            PKG_UPDATE="yum update -y"
            ;;
        *) 
            error "Unsupported distribution: $ID"
            ;;
    esac
else
    error "Cannot detect distribution"
fi

log "Detected distribution: $ID using $PKG_MGR"

echo "[1/10] Checking prerequisites..."

# Check for kubectl
if ! command -v kubectl &> /dev/null; then
    log "Installing kubectl..."
    if [[ "$PKG_MGR" == "apt" ]]; then
        sudo $PKG_UPDATE
        sudo $PKG_INSTALL curl apt-transport-https ca-certificates
        curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
        echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
        sudo $PKG_UPDATE
        sudo $PKG_INSTALL kubectl
    else
        sudo $PKG_INSTALL curl
        curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
        sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
        rm -f kubectl
    fi
fi

# Check kubectl access
if ! kubectl cluster-info &> /dev/null; then
    error "kubectl is not configured or cluster is not accessible"
fi

echo "[2/10] Installing Istio..."

# Download and install Istio
if [ ! -d "istio-"* ]; then
    curl -L https://istio.io/downloadIstio | sh -
fi

cd istio-*
export PATH=$PWD/bin:$PATH

# Install Istio with demo profile
istioctl install --set values.defaultRevision=default --set values.pilot.traceSampling=100.0 -y

echo "[3/10] Enabling automatic sidecar injection..."

kubectl label namespace $DEFAULT_NAMESPACE istio-injection=enabled --overwrite
kubectl get namespace $DEFAULT_NAMESPACE --show-labels

echo "[4/10] Creating Elasticsearch deployment..."

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: $ISTIO_NAMESPACE
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:$ELASTICSEARCH_VERSION
        env:
        - name: discovery.type
          value: single-node
        - name: ES_JAVA_OPTS
          value: "-Xms512m -Xmx512m"
        - name: xpack.security.enabled
          value: "false"
        ports:
        - containerPort: 9200
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1"
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: $ISTIO_NAMESPACE
spec:
  selector:
    app: elasticsearch
  ports:
  - port: 9200
    targetPort: 9200
EOF

echo "[5/10] Installing Prometheus..."
kubectl apply -f samples/addons/prometheus.yaml

echo "[6/10] Installing Grafana..."
kubectl apply -f samples/addons/grafana.yaml

echo "[7/10] Installing Jaeger with Elasticsearch backend..."

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: $ISTIO_NAMESPACE
  labels:
    app: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
      - name: jaeger
        image: jaegertracing/all-in-one:$JAEGER_VERSION
        env:
        - name: SPAN_STORAGE_TYPE
          value: "elasticsearch"
        - name: ES_SERVER_URLS
          value: "http://elasticsearch.$ISTIO_NAMESPACE.svc.cluster.local:9200"
        - name: ES_NUM_SHARDS
          value: "1"
        - name: ES_NUM_REPLICAS
          value: "0"
        ports:
        - containerPort: 16686
          protocol: TCP
        - containerPort: 14250
          protocol: TCP
        - containerPort: 14268
          protocol: TCP
        resources:
          requests:
            memory: "512Mi"
            cpu: "200m"
          limits:
            memory: "1Gi"
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger
  namespace: $ISTIO_NAMESPACE
spec:
  selector:
    app: jaeger
  ports:
  - name: query-http
    port: 16686
    targetPort: 16686
  - name: grpc
    port: 14250
    targetPort: 14250
  - name: http
    port: 14268
    targetPort: 14268
EOF

echo "[8/10] Installing Kiali..."
kubectl apply -f samples/addons/kiali.yaml

echo "[9/10] Configuring Istio telemetry..."

cat <<EOF | kubectl apply -f -
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: default
  namespace: $ISTIO_NAMESPACE
spec:
  accessLogging:
  - providers:
    - name: otel
EOF

echo "[10/10] Verifying installation..."

# Wait for deployments to be ready
log "Waiting for deployments to be ready..."
kubectl wait --for=condition=available --timeout=300s deployment/elasticsearch -n $ISTIO_NAMESPACE
kubectl wait --for=condition=available --timeout=300s deployment/jaeger -n $ISTIO_NAMESPACE
kubectl wait --for=condition=available --timeout=300s deployment/kiali -n $ISTIO_NAMESPACE
kubectl wait --for=condition=available --timeout=300s deployment/prometheus -n $ISTIO_NAMESPACE
kubectl wait --for=condition=available --timeout=300s deployment/grafana -n $ISTIO_NAMESPACE

# Verify services are running
log "Verifying services..."
kubectl get pods -n $ISTIO_NAMESPACE

log "Installation completed successfully!"
log ""
log "Access the observability tools using port-forwarding:"
log "  Kiali:      kubectl port-forward -n $ISTIO_NAMESPACE svc/kiali 20001:20001"
log "  Jaeger:     kubectl port-forward -n $ISTIO_NAMESPACE svc/jaeger 16686:16686"
log "  Grafana:    kubectl port-forward -n $ISTIO_NAMESPACE svc/grafana 3000:3000"
log "  Prometheus: kubectl port-forward -n $ISTIO_NAMESPACE svc/prometheus 9090:9090"
log ""
log "Then access:"
log "  Kiali:      http://localhost:20001"
log "  Jaeger:     http://localhost:16686"
log "  Grafana:    http://localhost:3000"
log "  Prometheus: http://localhost:9090"

Review the script before running. Execute with: bash install.sh

#istio #kubernetes #jaeger #kiali #observability

Implement Istio observability with Jaeger tracing and Kiali dashboard for Kubernetes service mesh

Prerequisites

What this solves

Prerequisites

Step-by-step installation

Install Istio with observability components

Enable automatic sidecar injection

Install observability add-ons

Configure Jaeger with Elasticsearch backend

Deploy Elasticsearch for Jaeger storage

Configure Istio telemetry settings

Configure Kiali with authentication

Deploy sample application for testing

Configure Prometheus service discovery

Set up port forwarding for dashboard access

Jaeger UI

Grafana dashboard

Prometheus UI

Generate traffic for observability testing

Generate traffic

Configure advanced observability features

Set up custom Grafana dashboards

Configure alerting rules

Verify your setup

Check Istio sidecar injection

Test Kiali access

Test Jaeger access

Check Prometheus targets

Configure production security

Enable mTLS for observability components

Configure network policies

Common issues

Next steps

Running this in production?

Related tutorials

Setup advanced Airflow alerting with Slack integration and custom notification rules

Configure Kubernetes ingress controller with NGINX and SSL certificates using cert-manager

Set up Kubernetes persistent volume snapshots and backup automation

Don't want to manage this yourself?