Observability cost is now the line item most engineering leaders cannot explain. The invoice arrives, it is thirty per cent larger than the quarter before, nobody deployed anything unusual, and the only honest answer available in the room is that a service somewhere started saying more than it used to. That is an uncomfortable position to defend, because the spend is real, the growth is compounding, and the thing being bought is genuinely valuable.
This guide takes the bill apart signal by signal. Logs, metrics and traces are priced on completely different mechanics, they fail expensively in completely different ways, and treating them as one telemetry budget is why blanket cuts either save nothing or remove the exact data an on-call engineer needed. You will find the four levers vendors actually charge on, a comparison of the common pricing models, benchmark ranges worth quoting, a fully worked observability cost model for a forty-service estate, and the controls that reduce spend without reducing coverage. It sits alongside our monitoring and cloud infrastructure practice.
It is written for teams running between twenty and a hundred services, because that is the band where the problem changes shape. Below it, the platform bill is small enough that nobody looks. Above it, there is usually a dedicated team and a chargeback model already. In between, the spend has become material, the ownership has not caught up, and observability is still treated as a tool someone bought rather than a data pipeline somebody has to run. If you have not yet measured your delivery baseline, our DevOps maturity assessment guide is the right place to start.
Table of contents
- Why observability cost became a board-level problem
- What actually drives observability cost
- Logs: the largest observability cost line
- Metrics: cardinality is the hidden observability cost
- Traces: sampling decides the observability cost curve
- Vendor pricing models and observability cost exposure
- Observability cost benchmarks worth quoting
- A worked observability cost model for a 40-service estate
- Cutting observability cost without losing incident coverage
- Retention tiers and archive: the cheapest observability cost lever
- Self-hosted versus SaaS observability cost
- OpenTelemetry, collectors and pipeline control
- Governance: making observability cost someone’s job
- Observability cost mistakes that survive review
- Frequently asked questions about observability cost
- References
Why observability cost became a board-level problem
Observability cost used to sit inside the infrastructure line and nobody separated it out. It is separated out now because it grew faster than the estate that produces it, and because the growth curve does not look like the one finance was shown at contract signature.
The bill grew faster than the estate
Container counts rose, but telemetry per container rose faster. A monolith emitted one log stream and a few dozen metrics. Forty services behind a service mesh emit forty log streams, sidecar logs, mesh access logs, per-pod metrics, per-route metrics and a trace for every request that crosses a boundary. The estate doubled and the telemetry went up sixfold, which is why the observability cost per host looks wrong to everybody who last priced it three years ago.
Three signals with three different price curves
Logs scale with how talkative your code is. Metrics scale with how many unique label combinations you emit, which is a multiplication problem rather than an addition one. Traces scale with request volume and with how much you attach to each span. One budget covering three curves means a saving in the cheap signal gets eaten by growth in the expensive one, and the observability cost total barely moves.
Nobody owns the meter
The team that generates the data is rarely the team that receives the invoice. A developer adding a debug line, a label or a new span attribute is making a purchasing decision without any price signal at all. That is not carelessness; it is a missing feedback loop, and it is the single most common structural cause of an observability cost problem.
What “too expensive” usually means
When a leadership team says observability is too expensive, they almost never mean the tool is bad value. They mean the spend is unpredictable, unattributable and rising without a decision. Predictability is worth more to a finance director than a discount, which is why the fix is usually governance and shaping rather than a renegotiation.
What actually drives observability cost
Every vendor prices on some combination of four levers. Once you can name which lever your invoice is dominated by, the remedy stops being guesswork. Most estates are dominated by one of them and pay only rounding on the other three.
Volume ingested, not volume stored
Almost all platforms charge on ingest, before compression, before filtering, before you decide whether the data was ever worth keeping. This matters enormously: dropping a field at query time saves nothing at all, while dropping it in the collector saves the whole line. Any observability cost work that starts at the storage tier has already missed the meter.
Cardinality and unique time series
A metric is not one thing. It is one series per unique combination of label values, so a single counter with a customer identifier attached to it can become tens of thousands of billable series overnight. Cardinality is multiplicative, arrives without warning, and is the reason a metrics invoice can triple in a week when the code change looked trivial in review.
Retention and query window
Retention is a straight multiplier on stored volume, and it is usually set once and never revisited. Ninety days of full-fidelity logs is a default in a great many contracts, and it is far longer than the window anybody actually queries. Retention is the least painful observability cost lever to pull because it changes nothing about what your engineers see this week.
Users, hosts and seats
Per-host and per-seat pricing feels predictable until autoscaling, ephemeral environments and short-lived batch nodes start minting hosts. Then a platform priced on hosts behaves like a platform priced on volume, without the visibility. Read how your contract counts a host, because the definition — and not the rate — is where the surprise lives.
| Billing lever | What it measures | How it grows | Fastest control | Typical share of bill |
|---|---|---|---|---|
| Ingest volume | Gigabytes accepted at the door | Linear with log verbosity and traffic | Filter and sample in the collector | 45–65% |
| Cardinality | Unique metric time series | Multiplicative with each new label | Drop or bucket high-variance labels | 10–30% |
| Retention | Days of queryable storage | Linear multiplier on stored volume | Tier by data set, archive the rest | 10–25% |
| Hosts and seats | Agents, nodes, named users | Step changes with autoscaling | Check the host definition in the contract | 5–20% |
| Egress and rehydration | Data pulled back out or restored | Spiky, incident-driven | Price it before you archive | 0–5% |
Logs: the largest observability cost line
In most estates logs are between half and two thirds of the observability cost, and they are also the signal with the largest proportion of data that is never read once. That combination makes them the correct place to start.
Debug logging left on in production
The classic pattern: an incident happens, someone raises the log level to debug to diagnose it, the incident closes, and the level stays. Six months later that service is emitting eleven times its normal volume and nobody has noticed because the change was invisible in the deployment record. Auditing log levels across every service is a one-afternoon job and it routinely finds a double-digit percentage of the entire bill.
The same line stored three times
A request gets logged by the application, again by the ingress controller, again by the service mesh sidecar and once more by the load balancer. Four copies of one event, each with its own overhead, all ingested and all billed. Deduplicating overlapping layers is unglamorous work with a very direct effect on observability cost, and it usually improves search quality at the same time.
Structured logs are cheaper than prose
A structured event with typed fields compresses better, filters better and can be aggregated into a metric rather than kept as raw text. Free-text log lines with a timestamp glued to a stack trace are expensive to store and almost impossible to filter safely. Moving to structured output is one of the few changes that lowers observability cost and raises usefulness at once.
What to drop, sample and keep
Health check chatter, successful readiness probes, framework startup banners and routine debug output can go entirely. High-volume success paths can be sampled at one in a hundred without losing the shape of the traffic. Errors, warnings, audit events, authentication decisions and anything with a compliance obligation are kept in full, always.
Metrics: cardinality is the hidden observability cost
Metrics look cheap. A counter is a number. The trouble is that the number is stored once per unique combination of labels, so the price is set by a decision made in a pull request that nobody reviewed for its billing consequences.
Why one label multiplies the bill
Take a request counter with labels for service, endpoint and status code: fifteen services, forty endpoints, six status codes gives 3,600 series, which is nothing. Add a customer identifier with two thousand values and it becomes 7.2 million. Nothing about the code looks different. The observability cost went up by three orders of magnitude, and it will keep going up as customers are added.
The labels that always cause trouble
User identifiers, session identifiers, request identifiers, full URL paths with embedded identifiers, container identifiers, pod names, build hashes and raw error strings. Each is unbounded or nearly so. Every one of them belongs on a log line or a span attribute, where it is stored once, rather than on a metric, where it is stored forever in every series it touches.
Histograms cost more than they look
A histogram is not one series, it is one series per bucket plus a sum and a count. A twelve-bucket latency histogram on a metric that already has 3,600 label combinations is over fifty thousand series on its own. Histograms are worth the money for latency objectives and rarely worth it anywhere else, so use them deliberately.
Pre-aggregate before you store
Recording rules and collector-side aggregation let you keep the dashboard you actually look at while discarding the raw dimensionality behind it. If a label is only ever used in a group-by that collapses it, aggregate it away at write time. This is the highest-yield metrics change available and it is invisible to every dashboard your team uses.
| Label added | Distinct values | Series before | Series after | Verdict |
|---|---|---|---|---|
| status_code | 6 | 600 | 3,600 | Keep — bounded and essential |
| deployment_region | 3 | 3,600 | 10,800 | Keep — small and decision-useful |
| customer_id | 2,000 | 3,600 | 7,200,000 | Drop — put it on a log or span |
| pod_name | ~400 per week | 3,600 | 1,440,000 | Drop — churns on every deploy |
| url_path (raw) | Unbounded | 3,600 | Unbounded | Normalise to a route template first |
Traces: sampling decides the observability cost curve
Tracing is the signal teams most often switch off first and regret second. It is also the signal where a single configuration value moves the observability cost by two orders of magnitude, which makes it the one worth understanding properly before you touch it.
Head sampling versus tail sampling
Head sampling decides at the start of a request whether to keep the trace, which is cheap to run and blind to what happened next. Tail sampling waits for the trace to complete and decides on the outcome, so it can keep every error and every slow request while discarding the fast successful majority. Tail sampling costs more compute in the pipeline and saves far more at the meter.
What a one per cent sample really buys
At one per cent you retain excellent statistical shape and almost no ability to find the specific failing request a customer just reported. That trade is fine for capacity work and poor for debugging. The usual answer is not a single rate: keep a low baseline for successful traffic and one hundred per cent of errors, slow requests and anything carrying a flag you set deliberately.
Keep the traces that matter
A sensible policy keeps every trace containing an error status, every trace above the latency objective, every trace touching a payment or authentication path, and a small percentage of everything else. That typically retains under five per cent of volume while keeping close to all of the diagnostic value, which is the best observability cost trade available anywhere in the stack.
Span attributes are logs in disguise
Teams that have shaped their logs carefully sometimes push the same verbosity onto spans instead, attaching full request bodies, headers and stack traces as attributes. Spans are billed on size as well as count. Attribute discipline deserves the same review as log discipline, or the saving simply moves from one line of the invoice to another.
Vendor pricing models and observability cost exposure
Pricing models are not interchangeable. The same estate can produce very different invoices depending on which model it is billed under, and the right model depends on the shape of your telemetry rather than on the headline rate.
| Pricing model | You pay for | Predictable when | Blows up when | Main defence |
|---|---|---|---|---|
| Per gigabyte ingested | Volume accepted, pre-compression | Log volume is shaped and stable | Debug level, retries, verbose libraries | Collector filtering and sampling |
| Per host or agent | Monitored nodes | Fleet size is stable | Autoscaling and ephemeral CI nodes | Agree the host definition in writing |
| Per time series | Unique metric label combinations | Labels are governed | One unbounded label ships | Cardinality limits and pre-aggregation |
| Per user or seat | Named logins | The audience is small | Read-only viewers get charged as editors | Tiered roles and dashboards on screens |
| Per query or scan | Data touched at read time | Dashboards are curated | Auto-refreshing wide-range dashboards | Cache, narrow ranges, kill idle refreshes |
| Self-hosted open source | Compute, storage and staff | You have platform capacity | The team that ran it leaves | Cost the people, not just the nodes |
Ingest pricing rewards shaping
Under per-gigabyte pricing every byte you stop at the collector is money saved at full rate, which makes pipeline work the highest-return activity available. It also means a single misconfigured library can cost thousands in a weekend, so ingest alerts belong on the platform itself.
Host pricing rewards fleet discipline
Per-host models look calm until an autoscaling group triples at peak or a build fleet spins up two hundred short-lived agents nightly. Ask precisely how a host is counted, whether it is peak or average, and whether containers count separately from nodes. The observability cost difference between those definitions can exceed the difference between vendors.
Series pricing rewards governance
If you are billed per time series, a cardinality budget per team is not bureaucracy, it is the product control. Enforce it at the collector so a bad label is rejected at source rather than discovered on an invoice five weeks later.
Everyone rewards a shorter retention default
Whatever the model, the retention default in the contract is usually longer than the window your engineers query. Shortening it is the least disruptive saving available, and it is worth doing before any negotiation so you are not buying capacity you were about to stop using.
Observability cost benchmarks worth quoting
Benchmarks are conversation starters rather than targets, but they are useful for one specific purpose: telling you whether you have a shaping problem or a scale problem. These are planning ranges we use with mid-sized engineering teams, not survey findings.
As a share of the cloud bill
Between five and fifteen per cent of total cloud spend is the range most teams land in once telemetry is shaped. Above twenty per cent, something is unshaped rather than expensive, and the fix is almost always in logs. Below three per cent, check that you have coverage at all before congratulating anyone.
Per engineer and per service
A useful sanity check is observability cost per engineer per month, which commonly falls between eighty and three hundred pounds in a shaped estate. Per service per month, forty to two hundred pounds is a reasonable band. A service well outside that band is usually one service, not a platform problem, and it can be found in an afternoon.
The ratio that actually matters
More telling than any absolute figure is the ratio of data ingested to data queried. If less than two per cent of ingested logs are ever touched by a query, you are running an archive with a search box attached, at premium prices. That single ratio justifies more shaping work than any benchmark table.
When the number stops being defensible
Spend becomes indefensible at the point where it is growing faster than traffic, faster than headcount and faster than revenue, with no decision behind the growth. That is the argument to bring to a budget conversation — not the absolute total, which is easy to defend, but the uncontrolled gradient, which is not.
A worked observability cost model for a 40-service estate
Numbers make the argument. This is a complete before-and-after model for a mid-sized platform, with every assumption stated so you can substitute your own and keep the structure.
The estate being modelled
Forty services on Kubernetes across three environments, 120 nodes at average load, roughly 900 requests per second at peak, 22 engineers with platform access, and a commercial contract billed on ingest with a ninety-day retention default. Nothing about it is unusual, which is the point.
Where the money goes today
Logs run at 1.1 terabytes a day across all environments, metrics at 1.4 million active series, traces at full capture on every request, and seats for all 22 engineers. The monthly invoice lands at just over £24,000, and the observability cost has risen for six consecutive months without a corresponding change in traffic.
The four changes applied
Non-production log levels dropped from debug to warn. Mesh and ingress logs deduplicated against application logs. Three unbounded metric labels removed and two histograms retired. Tail sampling introduced at five per cent baseline with full retention of errors and slow requests. Retention split into thirty days hot and eleven months in archive.
What the model produces
Log volume falls to 260 gigabytes a day, active series to 380,000, trace retention to roughly four per cent of spans. The monthly figure lands near £8,900, a reduction of about sixty-three per cent, with no reduction in error visibility and a small, deliberate loss of low-value success-path detail.
| Line | Before | After | Monthly before | Monthly after | Change applied |
|---|---|---|---|---|---|
| Logs | 1.1 TB/day | 260 GB/day | £13,900 | £3,300 | Levels, dedupe, sampling of success paths |
| Metrics | 1.4M series | 380K series | £5,300 | £1,650 | Three labels dropped, two histograms retired |
| Traces | 100% capture | ~4% retained | £3,100 | £1,150 | Tail sampling, errors and slow kept in full |
| Retention | 90 days hot | 30 hot + archive | £1,400 | £700 | Tiering, object-store archive |
| Seats and agents | 22 full seats | 8 full, 14 read | £900 | £480 | Role tiering |
| Pipeline compute | Minimal | Collector fleet | £0 | £1,620 | New cost, deliberately incurred |
| Total | — | — | £24,600 | £8,900 | 63% reduction |
Cutting observability cost without losing incident coverage
The reason cost programmes stall is fear, and the fear is rational. Somebody remembers the outage where the one log line that explained everything had been sampled away. The way through is to cut against evidence rather than against a percentage target.
Start from what you actually queried
Every platform can tell you which indexes, dashboards and log sources were touched in the last ninety days. Anything with zero reads in that window is a candidate, and the conversation with its owning team becomes concrete rather than theoretical. This one report reliably identifies the first third of any observability cost reduction.
Cut at the source, then the pipeline, then the store
Stopping data being emitted is the cheapest and most permanent fix. Filtering it in the collector is next, and still saves the full ingest rate. Reducing what you store saves the least, because ingest has already been charged. Working in that order is what separates a genuine observability cost reduction from a rearrangement.
Protect the golden signals absolutely
Latency, traffic, errors and saturation stay at full fidelity for every user-facing service, along with everything that feeds a service level objective or a page. Publishing that protected list before you start turns a cost programme from a threat into a negotiation with clear boundaries, which is why it belongs in the first conversation.
Measure the coverage you gave up
Record what was removed, and review it after the next two significant incidents. If a responder had to guess because data was missing, restore that source and take the saving elsewhere. A written record of removals also protects the programme politically, since it converts a vague suspicion into a specific, checkable claim. Our incident response practice covers what responders need to keep.
Retention tiers and archive: the cheapest observability cost lever
Retention is where the largest observability cost saving with the smallest behavioural change lives, and it is nearly always set to a contract default that no engineer chose. Splitting it by data set rather than applying one number to the platform is the whole technique.
Hot, warm and cold
Hot storage is fast, searchable and expensive, and it needs to cover the window in which people actually investigate — typically seven to thirty days. Warm storage covers the trend and capacity questions. Cold archive in object storage covers audit and compliance obligations at a small fraction of the price per gigabyte.
What compliance actually requires
Retention obligations usually apply to specific record types — authentication events, administrative actions, financial transactions, access to personal data — and not to every debug line your services produce. Reading the obligation properly often shortens the default dramatically for the ninety per cent of telemetry it never covered. Log management guidance is explicit that retention should follow the record type.
Rehydration is not free
Archived data has to come back before it can be searched, and that restore has both a cost and a delay. Price a realistic restore before designing the tiering, and make sure the delay is acceptable during an incident. An archive nobody can use in time is a compliance artefact, not an observability capability.
Set retention per data set
Audit logs might need seven years, application debug logs seven days, and trace data thirty. One retention number across all of them means you are paying audit-grade prices for debug output. Per-data-set retention is a configuration change with no code impact and a very direct observability cost effect.
| Data set | Hot | Warm | Archive | Why |
|---|---|---|---|---|
| Application debug logs | 3–7 days | None | None | Read within days or never read |
| Application error logs | 30 days | 90 days | 13 months | Trend analysis and recurrence checks |
| Access and audit events | 30 days | 12 months | 6–7 years | Regulatory and investigative need |
| Service level metrics | 30 days full | 13 months rolled up | 3 years rolled up | Year-on-year capacity planning |
| Traces | 7–14 days | None | Errors only, 90 days | Diagnostic value decays fast |
Self-hosted versus SaaS observability cost
The build-or-buy question reappears every renewal, and it is usually argued on licence price alone. That comparison is wrong in both directions, because it ignores the staff on one side and the flexibility on the other.
What self-hosting really costs
Compute, storage, object storage, cross-zone traffic, and then the part that decides the answer: the engineers who run it. Two competent platform engineers, partially allocated, is a realistic standing cost for a production-grade stack at this size. Include on-call for the telemetry platform itself, because an outage in the system that watches everything else is a bad night.
Where a managed platform wins
Correlation across signals, mature alerting, incident workflow integration, and not having to page anybody when the metrics store fills up. For teams without a platform function, the observability cost of a managed product is frequently lower than the true cost of the alternative once staff time is counted honestly.
The hybrid split that usually works
Self-host the high-volume, low-complexity signal — typically logs — and buy the correlation layer. Route with a vendor-neutral collector so the split can be revisited without re-instrumenting anything. This shape gives most of the saving and very little of the operational risk, and it is where a large share of mid-sized teams end up.
The exit question
Ask what leaving costs before you sign: how instrumentation is defined, whether dashboards and alert rules are exportable, and what happens to historical data. Vendor-neutral instrumentation is the cheapest insurance available against a renewal negotiation you cannot walk away from, and it is a cost optimisation decision as much as an architectural one.
| Factor | Self-hosted open source | Managed SaaS platform | Hybrid split |
|---|---|---|---|
| Marginal cost per extra GB | Low | High | Low for logs, high for the rest |
| Standing staff cost | 1.5–2 engineers | 0.2 engineer | 0.5–1 engineer |
| Time to first value | Weeks to months | Days | Weeks |
| Cross-signal correlation | You assemble it | Included and mature | Bought where it matters |
| Cost predictability | High — capacity based | Low — usage based | Medium |
| Switching difficulty | Low with open standards | High if instrumented natively | Low by design |
OpenTelemetry, collectors and pipeline control
The reason vendor-neutral instrumentation keeps appearing in observability cost conversations is that it converts a commercial problem into a configuration one. Once telemetry leaves your code in a standard format, where it goes and how much of it goes there become decisions you can change on a Tuesday.
Instrument once, route anywhere
Native agent instrumentation ties every service to one vendor’s data model, so changing supplier means touching every repository. A standard instrumentation layer means the routing decision moves into a collector configuration, which is precisely where you want a commercial decision to live.
Filter and transform in the collector
A collector can drop fields, redact personal data, normalise route templates, convert high-volume logs into counters and aggregate metrics before anything is billed. This is the single most valuable piece of infrastructure in an observability cost programme, and it is the one most estates do not yet run.
Tail sampling belongs in the pipeline
Because the collector sees complete traces, it is the right place to apply outcome-based sampling rules. Errors kept, slow requests kept, a small share of the rest kept, everything else dropped before the meter. Running this well requires capacity in the collector fleet, which is a real and worthwhile cost to add back.
The portability argument is commercial
Teams often present standard instrumentation as an architectural preference. It is better framed as leverage: a supplier who knows migration would take two quarters prices differently from one who knows it would take two weeks. That framing also travels better with a finance audience than a discussion of data models.
Governance: making observability cost someone's job
Every technical control on this page decays without an owner. Telemetry regrows, because adding a log line is easy, adding a label is easy, and neither has a price attached at the moment of the decision.
Show back before you charge back
Start by showing each team its own consumption monthly, with no money attached. Visibility alone typically removes a meaningful share of waste, because most excess is accidental rather than needed. Formal chargeback can follow once the numbers are trusted, and our cloud cost allocation guide covers how to set that up without starting an argument.
A budget per team and per signal
Give each team a monthly volume and cardinality allowance and let them spend it as they see fit. Teams make far better trade-offs than a central policy can, provided they can see the meter. This is standard FinOps practice applied to a data pipeline rather than to compute.
Guardrails in code, not in a wiki
Cardinality limits, ingest rate limits per source, and a pull-request check that flags new unbounded labels. A rule enforced by the platform survives staff changes; a rule written in a wiki page does not survive the first busy quarter, and observability cost discipline is exactly the kind of rule that erodes quietly.
The monthly review that works
Thirty minutes, one dashboard, three questions: what moved, who caused it, and was it worth it. No blame, no approval gate, just a standing forum where growth has to be explained. Teams running this review consistently report that the invoice stops surprising them, which was the actual objective all along.
Observability cost mistakes that survive review
These are the failure patterns that get through planning because each one sounds sensible in a meeting. They are worth naming explicitly before you start.
Cutting by percentage instead of by evidence
A directive to reduce telemetry by forty per cent produces uniform cuts across valuable and worthless data alike, because that is the only way to hit an arbitrary number quickly. Cut by query evidence and the same saving arrives without the coverage loss.
Removing what you never look at until you need it
Some data has near-zero read frequency and enormous value on the day it is read: authentication events, deployment records, security-relevant audit trails. Read frequency is the wrong metric for these. Keep them, archive them cheaply, and take the observability cost saving from success-path debug output instead. This is also where a cybersecurity review earns its place in the programme.
Buying a second tool to watch the first
Adding a telemetry pipeline product to control the spend of a telemetry platform is sometimes right and often a way of paying twice. Try the collector you already have, measure the result, and only buy the extra layer if the remaining problem justifies it.
Treating a discount as the plan
A renegotiated rate on unshaped data is a saving that decays as volume grows back, and it usually comes with a longer commitment. Shape first, then negotiate against the smaller, better-understood number. Vendors are also more flexible with a customer who can describe their own consumption precisely.
Forgetting the pipeline has a cost
Collectors, tail samplers and aggregation jobs consume compute and need running. Models that ignore this look better than reality and lose credibility at the first invoice. The worked model above adds £1,620 a month back for exactly this reason.
Frequently asked questions about observability cost
How much should observability cost?
Between five and fifteen per cent of your total cloud bill is the usual shaped range for observability cost. The more useful test is direction rather than level: if the figure is growing faster than traffic and headcount without a decision behind it, the level is not the problem.
Does OpenTelemetry reduce the bill by itself?
Not directly. It removes the instrumentation lock-in that stops you acting, and it puts a collector in the path where filtering, aggregation and sampling become possible. The saving comes from what you do in that collector, not from the standard itself.
Should we self-host to save money?
Only with a platform team that can carry it. Self-hosting moves cost from a licence to a payroll line and to an on-call rota. The hybrid split — self-hosted logs, bought correlation — captures most of the observability cost saving with much less operational exposure.
What is the single fastest saving?
Audit log levels across all non-production environments, then deduplicate overlapping log sources. Both are configuration changes, both can be done in a day, and together they routinely remove a quarter to a third of the total. Retention tiering is the fastest saving after that.
How do we stop the spend growing back?
Guardrails in the platform plus a monthly observability cost review with named owners. Every control here decays without one, because emitting more data is always the path of least resistance for a developer under deadline pressure and there is no price signal at the keyboard.
Does cutting telemetry increase incident risk?
Only if you cut without evidence. Protecting the golden signals, keeping all errors and audit events, and recording every removal for review after the next two incidents keeps the risk measurable and reversible. Teams that publish the protected list before starting rarely meet real resistance.
References
OpenTelemetry Observability Primer
OpenTelemetry Collector Documentation
Google SRE Book: Monitoring Distributed Systems
Google SRE Workbook: Alerting on SLOs
Google SRE Book: Availability Table
FinOps Foundation: What is FinOps
DORA: Software Delivery Metrics
Microsoft Azure Monitor Logs Cost Calculations and Options
Microsoft Azure Well-Architected Cost Optimisation Checklist
AWS Well-Architected Framework: Cost Optimization Pillar
NIST SP 800-92: Guide to Computer Security Log Management
NCSC Incident Management Collection
GOV.UK Service Manual: Monitoring the Status of Your Service
Google Cloud Architecture Framework
Prometheus: Metric and Label Naming Practices