Prometheus + Thanos: Long-Term Metrics Storage for Kubernetes

The Prometheus Scaling Problem

Prometheus is excellent for single-cluster monitoring. But at enterprise scale:

• No long-term storage — default retention is 15 days
• No global view — each Prometheus sees only its cluster
• No downsampling — raw data at full resolution forever
• Single point of failure — one instance per cluster

Thanos solves all four problems.

Thanos Architecture

┌─────────────────────────────────────────────────────┐
│                    Thanos Query                      │
│            (Global Query Frontend)                   │
└────────────────────────┬────────────────────────────┘
                         │
         ┌───────────────┼───────────────┐
         │               │               │
┌────────▼───────┐ ┌─────▼──────┐ ┌──────▼──────┐
│ Thanos Sidecar │ │   Thanos   │ │   Thanos   │
│ (Cluster 1)    │ │   Store    │ │   Sidecar  │
│                │ │  Gateway   │ │ (Cluster 2) │
└───────┬────────┘ └──────┬─────┘ └──────┬──────┘
        │                  │              │
┌───────▼────────┐  ┌─────▼──────┐ ┌─────▼──────┐
│  Prometheus 1  │  │ Object     │ │Prometheus 2│
│  (cluster-a)   │  │ Storage    │ │(cluster-b) │
└────────────────┘  │ (S3/GCS)   │ └────────────┘
                    └────────────┘

Installation (Helm)

# Prometheus with Thanos sidecar
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set prometheus.prometheusSpec.thanos.enabled=true \
  --set prometheus.prometheusSpec.thanos.objectStorageConfig.secretName=thanos-objstore \
  --set prometheus.prometheusSpec.retention=6h  # Short local retention

# Thanos components
helm install thanos bitnami/thanos \
  --namespace monitoring \
  --set query.enabled=true \
  --set storegateway.enabled=true \
  --set compactor.enabled=true \
  --set ruler.enabled=true \
  --set objstoreConfig=<config>

Object Storage Config

apiVersion: v1
kind: Secret
metadata:
  name: thanos-objstore
stringData:
  objstore.yml: |
    type: S3
    config:
      bucket: thanos-metrics
      endpoint: s3.eu-west-1.amazonaws.com
      region: eu-west-1
      access_key: ${AWS_ACCESS_KEY}
      secret_key: ${AWS_SECRET_KEY}

Downsampling

Thanos Compactor automatically reduces resolution:

Result: Store years of metrics at minimal cost.

Multi-Cluster Global View

Query across all clusters from a single Grafana:

# Grafana datasource pointing to Thanos Query
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
data:
  thanos.yaml: |
    apiVersion: 1
    datasources:
      - name: Thanos
        type: prometheus
        url: http://thanos-query.monitoring:9090
        access: proxy
        isDefault: true

Cost Optimization

S3 cost for 300GB: ~$7/month. Storing the same metrics in Prometheus local storage would require 3TB of expensive SSD.

When to Use Thanos vs Alternatives

Related Articles

• AI Observability on Kubernetes
• Velero: K8s Backup and DR
• Cilium: eBPF Networking