Configure advanced Jaeger sampling strategies for high-traffic environments

Verify Jaeger components

Check that your Jaeger collector and agent are running correctly before configuring sampling.

kubectl get pods -n jaeger-system
For non-Kubernetes deployments:
sudo systemctl status jaeger-collector
sudo systemctl status jaeger-agent

Install sampling configuration tools

Install curl and jq for testing sampling configurations and querying Jaeger APIs.

sudo apt update
sudo apt install -y curl jq

sudo dnf install -y curl jq

Sampling strategy types overview

Jaeger supports four main sampling strategies, each suited for different traffic patterns and observability requirements.

Check current sampling configuration

Query your Jaeger collector to see the current sampling strategy being used.

curl -s "http://jaeger-collector:14268/api/sampling?service=your-service-name" | jq .

Set up basic probabilistic sampling

Probabilistic sampling captures a fixed percentage of traces. This is ideal for high-volume services with consistent traffic patterns.

{
  "service_strategies": [
    {
      "service": "frontend-service",
      "type": "probabilistic",
      "param": 0.1
    },
    {
      "service": "payment-service",
      "type": "probabilistic",
      "param": 0.5
    }
  ],
  "default_strategy": {
    "type": "probabilistic",
    "param": 0.01
  }
}

Configure operation-level probabilistic sampling

Fine-tune sampling rates for specific operations within services to capture more data from critical endpoints.

{
  "service_strategies": [
    {
      "service": "api-gateway",
      "type": "probabilistic",
      "param": 0.05,
      "operation_strategies": [
        {
          "operation": "POST /api/v1/orders",
          "type": "probabilistic",
          "param": 1.0
        },
        {
          "operation": "GET /health",
          "type": "probabilistic",
          "param": 0.001
        }
      ]
    }
  ]
}

Configure rate limiting sampling

Rate limiting controls the absolute number of traces per second, protecting storage from traffic spikes while ensuring consistent data volume.

{
  "service_strategies": [
    {
      "service": "high-traffic-api",
      "type": "ratelimiting",
      "param": 100
    },
    {
      "service": "background-worker",
      "type": "ratelimiting",
      "param": 10
    }
  ],
  "default_strategy": {
    "type": "ratelimiting",
    "param": 50
  }
}

Combine rate limiting with probabilistic sampling

Use hybrid strategies to apply both probabilistic filtering and rate limits for maximum control over trace volume.

{
  "service_strategies": [
    {
      "service": "web-frontend",
      "type": "probabilistic",
      "param": 0.1,
      "max_traces_per_second": 200
    }
  ]
}

Enable adaptive sampling on Jaeger collector

Adaptive sampling automatically adjusts sampling rates based on traffic volume and storage capacity.

sampling:
  adaptive:
    max_traces_per_second: 1000
    strategies_reload_frequency: 1m
    aggregation_buckets: 10
    delay: 2m
    initial_sampling_probability: 0.001
    min_sampling_probability: 0.00001
    max_sampling_probability: 1.0

Configure adaptive sampling parameters

Set target trace volumes and adjustment intervals for different services based on their importance and traffic patterns.

{
  "strategies": [
    {
      "service": "payment-processor",
      "max_traces_per_second": 500,
      "lookback_period": "5m"
    },
    {
      "service": "user-auth",
      "max_traces_per_second": 200,
      "lookback_period": "2m"
    },
    {
      "service": "logging-service",
      "max_traces_per_second": 10,
      "lookback_period": "10m"
    }
  ],
  "default": {
    "max_traces_per_second": 100,
    "lookback_period": "5m"
  }
}

Start Jaeger collector with adaptive sampling

Restart the collector with adaptive sampling configuration enabled.

# For Kubernetes deployments
kubectl patch deployment jaeger-collector -n jaeger-system -p '{
  "spec": {
    "template": {
      "spec": {
        "containers": [{
          "name": "jaeger-collector",
          "args": [
            "--sampling.strategies-file=/etc/jaeger/adaptive-sampling.json",
            "--sampling.strategies-reload-frequency=60s"
          ]
        }]
      }
    }
  }
}'

For systemd deployments
sudo systemctl restart jaeger-collector

Deploy sampling strategy server

Set up a centralized sampling strategy server that can be updated without restarting services.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-sampling-server
  namespace: jaeger-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: jaeger-sampling-server
  template:
    metadata:
      labels:
        app: jaeger-sampling-server
    spec:
      containers:
      - name: sampling-server
        image: jaegertracing/jaeger-collector:latest
        command:
        - /go/bin/collector-linux
        args:
        - --sampling.strategies-file=/etc/sampling/strategies.json
        - --sampling.strategies-reload-frequency=30s
        - --http.host-port=:14269
        ports:
        - containerPort: 14269
        volumeMounts:
        - name: sampling-config
          mountPath: /etc/sampling
      volumes:
      - name: sampling-config
        configMap:
          name: sampling-strategies

Create sampling strategy ConfigMap

Store sampling strategies in a ConfigMap that can be updated dynamically without container restarts.

kubectl create configmap sampling-strategies -n jaeger-system --from-file=/etc/jaeger/sampling-strategies.json

Apply the deployment
kubectl apply -f /etc/jaeger/sampling-server.yaml

Configure applications for remote sampling

Update your application configuration to use the remote sampling server instead of local configuration.

JAEGER_SAMPLER_TYPE=remote
JAEGER_SAMPLER_MANAGER_HOST_PORT=jaeger-sampling-server.jaeger-system.svc.cluster.local:14269
JAEGER_SAMPLER_REFRESH_INTERVAL=60

Set up sampling metrics monitoring

Configure Prometheus to collect sampling metrics from Jaeger components for performance monitoring.

- job_name: 'jaeger-sampling'
  static_configs:
  - targets: ['jaeger-collector:14269', 'jaeger-agent:14271']
  metrics_path: /metrics
  scrape_interval: 30s
  scrape_timeout: 10s

Create sampling performance alerts

Set up alerts for sampling rate anomalies and storage pressure indicators.

groups:
name: jaeger_sampling
  rules:
  - alert: JaegerSamplingRateHigh
    expr: jaeger_agent_sampling_rate > 0.5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Jaeger sampling rate is unusually high"
      description: "Service {{ $labels.service }} has sampling rate {{ $value }} which may indicate traffic anomaly"

  - alert: JaegerTraceVolumeHigh
    expr: rate(jaeger_collector_traces_received_total[5m]) > 10000
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Jaeger receiving excessive trace volume"
      description: "Collector is receiving {{ $value }} traces per second"

Update sampling strategies based on metrics

Use sampling metrics to dynamically adjust strategy parameters for optimal performance.

# Query current sampling rates
curl -s "http://jaeger-query:16686/api/services" | jq '.data[]'

Check trace volume by service
curl -s "http://prometheus:9090/api/v1/query?query=rate(jaeger_collector_traces_received_total[1h])" | jq '.data.result'

Update sampling configuration
kubectl patch configmap sampling-strategies -n jaeger-system --type merge -p '{
  "data": {
    "strategies.json": "{
      \"service_strategies\": [
        {
          \"service\": \"high-volume-service\",
          \"type\": \"probabilistic\",
          \"param\": 0.01
        }
      ]
    }"
  }
}'