Design a Metrics and Logging Service

You're asked to design the observability platform every other team at the company depends on: the system that collects metrics and log lines from thousands of services, makes them searchable within seconds, powers dashboards and alerts, and keeps enough history to investigate an incident from last quarter. Engineers should be able to ask "show me every error from the checkout service in the last hour, grouped by host" and get an answer before they lose their train of thought.

Metrics and logs arrive through the same agents, but they have different shapes and costs. Metrics are small, numeric, and predictable. Logs are large, unstructured, and bursty. Let's keep their ingestion shared but give them separate processing and storage paths.

Clarify the requirements

• Metrics, logs, or both? If both are in scope, establish early that they need separate storage.
• How fresh does the data need to be? Seconds for alerting, and minutes is acceptable for dashboards. That difference is what justifies two paths through the pipeline.
• How long is the retention? Days at full fidelity and months in cheaper storage is typical, and the answer drives most of the cost.
• Who queries it, and how? Free-text search over logs, aggregation over metrics, and alert rules evaluated continuously. Three different query shapes.
• What happens if the pipeline falls behind? Dropping data during an incident is exactly when you least want to, so ask what the buffering expectation is.

Assume: both data types, seconds of freshness for alerts, 30 days hot and 13 months cold, and buffering that survives a multi-hour outage.

Back-of-envelope numbers

• Log volume: $10\text{k hosts} \times 100 \text{ lines/sec} = 1\text{M log lines/sec}$
• Log bytes: $1\text{M/sec} \times 500\text{ B} = 500\text{ MB/sec} \approx 43\text{ TB/day}$ raw
• After compression at roughly $10\times$: $\approx 4\text{ TB/day}$, or $120\text{ TB}$ for 30 days hot
• Metrics: $10\text{k hosts} \times 200 \text{ series} \div 15\text{ sec} \approx 130\text{k points/sec}$, which compresses to a few hundred gigabytes a month
• Buffer for a 4-hour outage: $500\text{ MB/sec} \times 14{,}400\text{ sec} \approx 7\text{ TB}$

Logs are roughly two orders of magnitude more expensive than metrics. Use that estimate to justify indexing only part of the log stream and moving older data to cheaper storage.

High-level architecture

Components

1. Ingest tier. Authenticates agents, validates payloads, and enforces per-tenant quotas before anything is accepted.
2. Kafka buffer. Absorbs bursts and decouples ingestion from processing, so a slow index never rejects incoming data.
3. Metrics processor. Aggregates and writes numeric series.
4. Log processor. Parses, enriches, samples, and routes log lines.
5. Time series store. Compressed numeric points, partitioned by series and time.
6. Search index. Inverted index over recent logs, which is what makes free-text queries fast.
7. Object storage. The raw compressed log bodies, cheap and long-lived.
8. Alert evaluator. Runs rules continuously against recent metrics.
9. Query service. Fans a user query to whichever store can answer it.

Deep dive 1: buffering the ingest stream

Everything upstream of the buffer is a firehose you don't control, and everything downstream is a system with finite capacity. Putting a durable log between them is what keeps a slow consumer from turning into data loss.

Concretely, the buffer gives you four things:

• Burst absorption. An incident produces a log spike exactly when the data matters most, and the queue takes it while processing catches up.
• Independent consumers. Metrics and logs read the same stream at their own pace, and adding a third consumer later costs nothing upstream.
• Replay. A bug in the log parser is fixable by resetting the consumer offset and reprocessing, rather than by losing a day of data.
• Backpressure that stops at the right place. When the index is overwhelmed, consumers fall behind and the queue grows. Agents keep shipping successfully, which is what you want, because the alternative is agents buffering on production hosts and eventually filling their disks.

Size the buffer from the outage the system must survive. Use the seven-terabyte estimate above to turn "we'll use Kafka" into a concrete capacity decision.

Deep dive 2: storing metrics and logs differently

Do not put both data types in one store. Their shapes, query patterns, volume, and retention needs all differ.

Metrics go to a time series store, where points for one series are contiguous and compress to a byte or two each. The mechanics of that store are their own design question.

Logs need full-text search, which means an inverted index mapping terms to the documents containing them. The index is what costs money, so the design lever is what you put in it: index the structured fields of service, host, level, trace ID, and timestamp, plus the message body, but keep the raw line in object storage rather than duplicating everything into the index.

That split lets you tier honestly. Recent logs are indexed and instantly searchable. Older logs live only in object storage, where a query is a slower scan rather than an index lookup, at a small fraction of the cost.

Deep dive 3: controlling what you keep

At a terabyte an hour, the design question isn't how to store everything, it's how to decide what's worth storing. Four levers, in the order you'd apply them:

• Sampling by level. Keep every error and warning, and sample info and debug at some percentage. Most log volume is routine success messages that nobody has ever searched for.
• Aggregation at the agent. A thousand identical lines become one line with a count. This is enormously effective on tight loops and retry storms, which are exactly what produce volume spikes.
• Tiered retention. Full-fidelity indexed logs for days, then index-free object storage for months, then delete. Each tier is roughly an order of magnitude cheaper than the one above it.
• Per-tenant quotas. One team turning on debug logging shouldn't degrade observability for everyone else, so quotas belong at the ingest tier where they can reject rather than downstream where the damage is already done.

Structured logging makes these controls possible. A JSON line with consistent field names can be sampled by level, aggregated by template, and queried by field. Free text supports far fewer controls. Make structured logging a platform requirement because producers must adopt it before the data arrives.

Deep dive 4: alerting on a stream that can lag

Alerts are the part of this system that has to work when everything else is on fire, and that constraint shapes their design.

Evaluate alerts on the streaming path, not on the stored data. Reading from the buffer directly keeps alert latency at seconds even when the indexing pipeline is minutes behind, which is the state the system will be in during exactly the incident the alert exists to catch.

Absence of data is itself an alert condition. A service that stops reporting looks identical to a healthy service with nothing to say, and the difference matters enormously. Alert rules need a "no data received" case rather than only thresholds on values.

Deduplicate and group before notifying. One bad deploy across five hundred hosts produces five hundred firing rules, and paging someone five hundred times is the same as paging them zero times. Group by the alert rule and the smallest common dimension, and send one notification with a count.

State the pipeline's own health as a first-class metric. Consumer lag on the buffer, ingest rejection rate, and indexing latency should be monitored by something outside this system, since an observability platform that goes down silently takes every other team's visibility with it.

Common pitfalls

• One store for metrics and logs. Their volume, shape, and query patterns all differ by orders of magnitude.
• No buffer between ingestion and processing. A slow index becomes dropped data during the incident you most need to see.
• Indexing every field of every log line. The index becomes larger than the data and dominates the cost.
• Alerting off the indexed store. Alert latency inherits pipeline lag, which is worst exactly when it matters.
• No per-tenant quota. One team's debug flag degrades observability for the entire company.

Leveling signals

Related lessons

Design a Time Series Metrics StoreHard

Build the storage engine behind a monitoring system, with compression, cardinality limits, and rollups.

Design an Application Performance Monitoring SystemHard

Trace a single request across dozens of services and find where the latency went.