Datadog Custom Metric Billing Shock: Why Platform Teams Are Migrating to the Grafana LGTM Stack and SigNoz

The Anatomy of an Observability Invoice Shock

In late 2026, cloud observability expenditure has escalated into one of the top three unbudgeted infrastructure costs for cloud-native enterprises. Operating modern microservices on Kubernetes across dynamic multi-cloud environments generates vast volumes of telemetry: structured logs, Prometheus metrics, and distributed OpenTelemetry traces.

While commercial SaaS observability platforms like Datadog, New Relic, and Dynatrace provide polished dashboards and turnkey agent installation, their pricing models contain severe economic traps. Chief among these traps is the custom metric cardinality multiplier:

A single metric name (e.g., http_requests_total) with 5 user labels (such as user_id, region, service, http_status, and device_type) can easily spawn millions of unique time-series combinations.
At typical commercial rates of $5.00 per 100 custom metrics per month, an unexpected code deployment introducing a high-cardinality tag can trigger an overnight invoice escalation from $4,000/month to over $35,000/month.
Additional charges for log ingestion, indexing tiers, retention windows, and APM trace spans frequently result in observability costs that exceed the underlying compute infrastructure bills they are supposed to monitor.

In response, platform engineering and Site Reliability Engineering (SRE) teams are executing structured migrations toward open-source, standards-based observability architectures. Leading the charge are the Grafana LGTM Stack (Loki, Grafana, Tempo, Mimir) and SigNoz, both built upon the universal OpenTelemetry (OTel) telemetry collection standard.

---

Architectural Comparison: Datadog vs. Grafana LGTM vs. SigNoz

| :--- | :--- | :--- | :--- |

| Cardinality Penalty | Extreme (Overage charges per custom metric) | Managed via Mimir horizontally scalable compaction | Highly resilient (ClickHouse columnar compression) |

---

1. The Grafana LGTM Stack: Cloud-Native Object Storage Mastery

The brilliance of the modern Grafana LGTM stack lies in its radical reduction of operational storage costs. Rather than maintaining massive, expensive distributed search clusters (like Elasticsearch or OpenSearch), each component of the LGTM stack is purpose-built to leverage dirt-cheap cloud object storage (such as AWS S3, Google Cloud Storage, or MinIO):

Loki (Logging): Unlike Elasticsearch, Loki does not build a full-text index across every word in every log line. Instead, it only indexes container metadata labels (e.g., app, namespace, environment) and stores raw compressed log chunks directly in S3. Log volume queries leverage parallelized stream parsing (LogQL), slashing storage overhead by up to 90%.
Tempo (Distributed Tracing): Tempo achieves massive tracing scalability by storing trace spans in object storage indexed purely by Trace ID. By decoupling search indexing from trace retention, teams can retain 100% of user request traces for weeks at a fraction of commercial APM cost.
Mimir (Metrics): Mimir provides horizontally scalable, globally distributed Prometheus metric long-term storage with lightning-fast query execution, automatic multi-tenancy, and seamless compaction across billions of active series.
Grafana (Visualization): The de facto industry standard interface unifying metrics, logs, and traces into intuitive, correlated drill-down dashboards.

+-------------------------------------------------------------+
|                 GRAFANA LGTM STACK PIPELINE                 |
|                                                             |
|  [ Microservices ] ---> [ OpenTelemetry Collector ]         |
|                                    |                        |
|       +----------------------------+-------------------+    |
|       | (Metrics)                  | (Logs)            |    |
|       v                            v                   v    |
|  [ Grafana Mimir ]          [ Grafana Loki ]   [ Grafana Tempo ]
|       |                            |                   |    |
|       +----------------------------+-------------------+    |
|                                    |                        |
|                                    v                        |
|                      [ Low-Cost Object Storage (S3) ]       |
|                                    |                        |
|                                    v                        |
|                     [ Unified Grafana Dashboards ]          |
+-------------------------------------------------------------+

---

2. SigNoz: The ClickHouse-Powered All-in-One Powerhouse

While the LGTM stack provides unmatched modular scalability, configuring four separate backend services (Loki, Tempo, Mimir, Grafana) requires dedicated platform engineering maintenance. For teams seeking a turnkey single-pane-of-glass experience that closely mimics Datadog without the cost, SigNoz has emerged as the premier open-source solution.

Architectural Advantages of SigNoz:

Unified ClickHouse Backend: Metrics, logs, and distributed traces all reside within a unified ClickHouse analytical database. ClickHouse delivers blistering query performance, sub-second aggregation across billions of rows, and exceptional 4x to 8x data compression ratios.
Pure OpenTelemetry Native: SigNoz uses the OpenTelemetry collector as its native ingest mechanism. There are zero vendor-specific SDKs; if you ever decide to switch platforms in the future, your application instrumentation code remains 100% unchanged.
Turnkey APM Metrics: Automatically derives RED metrics (Rate, Errors, Duration), p95/p99 latency percentiles, and service dependency maps directly from incoming trace streams without extra configuration.

Deploying SigNoz via Docker Compose in 2 Minutes:

# Clone the official SigNoz deployment repository
git clone -b main https://github.com/SigNoz/signoz.git
cd signoz/deploy/docker

# Launch the unified SigNoz stack (ClickHouse, OTel Collector, Frontend)
docker compose up -d

# Verify services on http://localhost:3301

---

Mastering Cardinality Control: OpenTelemetry Pipeline Filtering

In high-scale Kubernetes clusters, preventing high-cardinality data from entering your backend is essential. The OpenTelemetry Collector provides granular filtering rules that sanitize telemetry at the edge:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 10000
    timeout: 10s
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20

  # Cardinality scrubbing processor
  attributes/filter:
    actions:
      - key: http.client_ip
        action: delete
      - key: user.email
        action: hash
      - key: ephemeral_container_id
        action: delete

  # Tail-based sampling: Retain 100% of errors and slow requests, sample 1% of normal 200 OKs
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 2000
    policies:
      - name: drop-healthy-http
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 1500 }
      - name: probabilistic-sample
        type: probabilistic
        probabilistic: { sampling_percentage: 1.0 }

exporters:
  otlp/signoz:
    endpoint: "signoz-otel-collector:4317"
    tls:
      insecure: true
  otlp/lgtm:
    endpoint: "tempo:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/filter, tail_sampling, batch]
      exporters: [otlp/signoz, otlp/lgtm]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, attributes/filter, batch]
      exporters: [otlp/signoz]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, attributes/filter, batch]
      exporters: [otlp/signoz]

---

Kubernetes DaemonSet Deployment Pattern with Helm

Deploying the OpenTelemetry Collector across a production Kubernetes cluster is most effectively managed via the official OpenTelemetry Helm chart configured in DaemonSet mode. This ensures that every worker node hosts a local collector pod, minimizing cross-node network hops and allowing applications to emit telemetry over local localhost:4317 gRPC sockets:

# Add the OpenTelemetry Helm repository
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

# Install the collector in DaemonSet mode
helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace observability \
  --create-namespace \
  --set mode=daemonset \
  --set presets.kubernetesAttributes.enabled=true \
  --set presets.kubeletMetrics.enabled=true \
  --values otel-collector-config.yaml

When deployed in this topology, the collector automatically enriches telemetry with Kubernetes pod names, namespace labels, node affinities, and deployment revision numbers without requiring application containers to query the Kubernetes API server directly.

---

Synthetic Alerting Rules for Prometheus and Grafana Mimir

A major concern for engineering teams transitioning away from Datadog Monitors is recreating proactive alerting rules. Grafana Mimir natively evaluates Prometheus recording and alerting rules with zero performance penalty.

Here is a production-grade alerting rule definition monitoring multi-service API failure spikes:

# alerting-rules.yaml
groups:
  - name: api-slos
    rules:
      - alert: HighAPIErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) * 100 > 2.5
        for: 3m
        labels:
          severity: critical
          team: platform-core
        annotations:
          summary: "API Error rate exceeds 2.5% threshold"
          description: "Cluster {{ $labels.cluster }} is experiencing elevated 5xx error responses (current: {{ $value }}%)."
          runbook_url: "https://wiki.internal/runbooks/high-error-rate"

These rules evaluate continuously inside Mimir's ruler component and dispatch instant webhook alerts to PagerDuty or Slack, guaranteeing 99.99% SLO visibility without proprietary monitoring fees.

---

Cold Storage Tiering: S3 Glacier Lifecycle for Long-Term Log Compliance

In heavily regulated industries (FINRA, HIPAA, PCI-DSS), retaining audit logs for 1 to 7 years is legally mandated. Under Datadog, retaining archived logs incurs persistent indexing re-hydration surcharges.

With Grafana Loki and SigNoz, long-term compliance is virtually free through automated S3 lifecycle rules:

Hot Tier (0–14 Days): Stored on NVMe SSDs or AWS S3 Standard for instant sub-second interactive querying by developers debugging active production releases.
Warm Tier (15–90 Days): Automated S3 Standard-Infrequent Access (S3-IA) transition ($0.0125/GB/month), reducing storage costs by 45%.
Cold Archive Tier (91–2,555 Days): Automated transition to AWS S3 Glacier Flexible Retrieval ($0.0036/GB/month). For a company generating 50 TB of logs annually, 7-year Glacier compliance storage costs less than $180 per month—compared to tens of thousands of dollars under proprietary SaaS tiers.

---

Real-World Financial Case Study: Fintech Scale-up

Organization Profile:

Workload: 180 microservices running on 420 EKS nodes.
Telemetry Volume: 2.8 TB logs/day, 4.5 billion trace spans/month, 85,000 custom metric series.

1. Previous Monthly Datadog Expenditure:

Infrastructure Host Licensing: 420 hosts × $15 = $6,300
APM Host Licensing: 420 hosts × $31 = $13,020
Ingested Logs: 2.8 TB/day × 30 days = 84 TB × $0.10/GB = $8,400
Indexed Logs (15-day retention): 12 TB × $1.70/million = $6,800
Custom Metrics Surcharges (Cardinality bursts): ~$8,500
Total Datadog Monthly Invoice: $43,020 / month ($516,240 / year)

2. Post-Migration Open-Source Stack Expenditure (SigNoz on AWS):

Compute Cluster: 3 × c6i.4xlarge EC2 instances = $1,470 / month
ClickHouse Storage: 12 TB gp3 EBS + S3 Cold Tiering = $680 / month
Network Data Transfer: ~$350 / month
Dedicated SRE Maintenance Allocation: ~$3,500 / month
Total Self-Hosted Monthly Cost: $6,000 / month ($72,000 / year)

Net Annual Financial Savings: $444,240 (86.1% Total Reduction)

---

Frequently Asked Questions (FAQ)

What happens to Datadog's Synthetic Monitoring and RUM?

Open-source solutions like Grafana Faro provide modern Real User Monitoring (RUM) for web frontend applications. For synthetic testing, open-source tools like Playwright running on scheduled Kubernetes CronJobs or Uptime Kuma provide superior scriptable user simulation without SaaS fees.

Is OpenTelemetry stable enough for production enterprise workloads?

Yes. OpenTelemetry is the second most active project in the entire CNCF ecosystem behind Kubernetes itself. The tracing, metrics, and logging specifications are officially 100% stable, backed by Google, Microsoft, AWS, Red Hat, and Cisco.

How do we handle alerting without Datadog Monitors?

Grafana Alerting provides advanced alerting rules evaluated against PromQL and LogQL expressions, routing notifications directly to PagerDuty, Slack, Opsgenie, and custom webhooks with built-in deduplication and quiet hours.

Can SigNoz or Grafana correlate logs with traces automatically?

Yes. OpenTelemetry automatically injects trace_id and span_id headers into application log lines. In both Grafana and SigNoz, clicking on an error log line instantly opens the exact distributed trace diagram showing which database call or downstream microservice threw the exception.

---

Strategic Verdict

Observability is vital for system reliability, but paying commercial vendors exponential markups on raw telemetry data is unsustainable. By adopting the Grafana LGTM Stack or SigNoz underpinned by the vendor-neutral OpenTelemetry standard, platform teams can cut observability budgets by 70% to 85% while gaining total architectural sovereignty.