When Two Workflows Share a Baseline but One Has Hidden Entropy

It started with a routine deployment. The team at a mid-size e-commerce platform had shared a p99 latency baseline across two workflows: payment processing and inventory checks. Both had used the same 200ms threshold for months. Then payments started timing out—intermittently, in bursts. Inventory checks stayed green. The baseline was identical, but one workflow had been accumulating entropy: larger payloads from a new payment gateway, slower database replicas, and a caching layer that stopped warming properly. No one noticed because the other workflow masked the average.

The Real Field: Where Shared Baselines Go Wrong

How a shared baseline hides tail latency shifts

The first time I watched a shared baseline fail, I was staring at a dashboard that looked perfectly healthy. P99 latency for the combined workflow sat at 210ms — well under our agreed 300ms ceiling. The on-call engineer shrugged. We almost closed the ticket. But the payment team was getting complaints. Customers saw spinning spinners on checkout, then timeouts. The dashboard said everything was fine. That gap — between what the baseline reports and what one workflow actually feels — is where hidden entropy lives. The aggregate number smooths the spike into invisibility. One workflow's tail creeps up by 80ms; the other workflow drops by 50ms after a deploy. The mean stays flat. The baseline stays green. Nobody panics. That's the trap.

‘A shared baseline is a contract between two workflows — but one of them can break the deal without telling the other.’

— site reliability engineer, after a postmortem that blamed ‘infrastructure noise’ for three weeks

The field dynamics that lead teams into this mess are surprisingly mundane. Most squads share a baseline because it feels efficient — one chart, one alert, one conversation. The payment team and the inventory team deploy to the same cluster, read from the same metrics pipeline, and use the same SLO dashboard. Why split? The answer comes later, during the incident. The inventory workflow is bursty but forgiving — it handles spikes in search traffic by queuing writes, so its latency distribution is wide but shallow. The payment workflow is the opposite: tight distribution, no buffer, hard real-time expectations. Same baseline, different entropy profiles. When inventory gets noisy — a cache stampede, a slow replica — its latency variance balloons. The payment workflow absorbs none of that variance, but the baseline still blends both. The payment team's p99 drifts upward by 40ms over two weeks. Nobody notices because the combined p99 stays under threshold. Then the drift hits a tipping point. Checkout fails. Returns spike. The shared baseline didn't lie — it just told the story of the average, not the victim.

The payment vs. inventory case study

Concrete example: a mid-market e-commerce platform I worked with. Two workflows sharing one baseline — payment authorization and inventory reservation. The inventory workflow ran a background job that reserved stock after payment succeeded. Responsive but not latency-critical. The payment workflow had a hard timeout of two seconds. Same baseline. Same alert. The inventory team optimized for throughput — they batched writes and accepted occasional latency spikes during reindex operations. The payment team optimized for tail consistency. For six months the baseline looked fine. Then a new inventory indexing job hit prod. It added 150ms to the inventory p99 during peak hours. The payment p99 climbed by 90ms — invisible in the combined metric. The team only caught it when the payment timeout hit 1.98 seconds during Black Friday traffic and the checkout failure rate jumped from 0.3% to 4.1%. The shared baseline had masked the drift for eight weeks. Worth flagging—the inventory team had no idea their deploy caused the problem. Their own p99 was still under threshold. The entropy was invisible to both sides until the seam blew out.

Most teams share baselines because splitting feels like overhead. Two dashboards, two alert policies, two on-call runbooks. That cost is real. But the hidden entropy cost — a degraded checkout path, a silent regression that compounds over weeks — is worse. The trick is knowing when the cost of sharing exceeds the cost of splitting. The answer is almost always: when one workflow has tight latency constraints and the other doesn't. That's the field where shared baselines go wrong. Not all shared baselines fail. But the ones that do fail exactly here — at the asymmetry of entropy, masked by the comfort of a single number.

Foundations People Get Wrong: Baseline ≠ Threshold

Static vs. dynamic baselines

Most teams treat a baseline like a painted line on a warehouse floor — fixed, permanent, safe to ignore. That works until the floor shifts. I have watched engineering teams hard-code a baseline for two workflows sharing one system: a daily batch job and a real-time API. The batch job ran at night, the API spiked at noon. They shared memory, disk, and a single baseline for p99 latency. Wrong order. The static number fit neither window — it was too loose for the API's quiet hours and too tight for the batch job's burst. A baseline is not a rock; it's a rubber band that weather and weight stretch differently. The catch is that dynamic baselines cost more to compute. You need rolling windows, seasonal adjustments, and a tolerance for occasional recalibration pain. Skip that cost and you get a number that technically exists but practically lies.

Signal vs. noise: entropy is not random

Here is where the confusion bites hardest. Entropy in observability is usually described as chaos — random jitter, unpredictable spikes, gremlins in the wire. That's wrong. Hidden entropy in a shared baseline is patterned but invisible because the wrong aggregation masks it. One workflow might emit a periodic heartbeat every 30 seconds; the other workflow emits a 200-millisecond burst when a user uploads a photo. Average them together and you get a smooth, harmless line. That smooth line is the trap. You lose the very signal that tells you something is degrading. The burst gets buried, the heartbeat gets flattened, and when both workflows degrade simultaneously — because one starved the other of CPU — your baseline looks fine right up until the page goes red. I fixed this once by splitting the metric stream before any baseline calculation. Ugly fix. Necessary fix.

'A baseline that hides one workflow's entropy is not a baseline. It's a shared hallucination.'

— senior SRE, postmortem notes, 2023

That sounds harsh but it's accurate. Most teams I see debug this miss the entropy because they look at the wrong percentile. The p50 shows calm. The p99 shows spikes but the team dismisses them as "the noisy workflow." They never ask: Is that noise actually a signal from the quieter workflow being trampled? One rhetorical question, but it saves a week of war-room meetings.

Why 'baseline' is often a misnomer in shared contexts

The word baseline implies a foundation — something you build on, trust, and rarely question. In a shared observability context, that implication is dangerous. What teams actually create is a blend, an average, a composite that belongs to no single workflow. Calling it a baseline gives it authority it doesn't deserve. The pitfall is that engineers start treating deviations from the blend as anomalies, when really the blend itself is the anomaly — it describes a workload that never actually runs. I have seen on-call rotations waste days chasing "baseline violations" that were just one workflow behaving normally while the other stuttered. Trade-off: you can keep the shared baseline for convenience (less data, fewer dashboards) but you must annotate every alert with "this might be workflow A, not B". Or you split early, pay the compute cost, and stop guessing. What usually breaks first is trust — once the team stops believing the baseline, they ignore it, and then you have no baseline at all. That hurts more than the split ever did. Pick your pain: the cost of separation now or the cost of hallucination later.

Honestly — most data posts skip this.

Patterns That Usually Work

Segmenting baselines by workflow type

The first pattern that actually holds up in production is brutally simple: build one baseline per logical workflow, not per service. I have watched teams shove every API call into a single 'latency' baseline because the endpoints live under the same deployment. That hides entropy—the fast read-path drowns out the slow write-path until the write-path silently triples its tail latency. The fix: separate baselines for 'user search' versus 'batch export', even if they share a database. Each gets its own mean, its own variance, its own seasonal window. The cost is more dashboards to maintain. The payoff is you spot that one workflow degrading before your pager goes off at 3 AM.

Sliding window baselines with adaptive thresholds

Static thresholds are a trap—they age like milk. A better pattern: sliding window baselines that recompute every hour using the last 7 days of data. The window slides, not jumps. Most teams skip this: they set a static 200ms threshold in January, and by July the system has changed three times and that threshold is pure fiction. Adaptive baselines recalculate the expected range from recent history—mean plus 2.5 standard deviations, or a 95th percentile band—and flag only when the current window deviates beyond that band. The catch is choosing the window size right. Too short (2 hours) and you normalize anomalies into the baseline. Too long (30 days) and you miss drift until it's a crisis. I have found 7 days with a 1-hour step works for most web-serving workloads. Worth flagging—this pattern fails during big launches or code freezes, because the window includes data from a different system state. You need a manual override for those weeks. That hurts, but less than paging your team on a false alarm every deploy.

'A baseline that adapts to last week's patterns will still miss a slow degradation that compounds over three weeks—you need a second, slower baseline for that.'

— senior SRE, after a 45-minute outage caused by 0.5% daily latency creep

Using percentiles and distribution analysis

Averages lie. The mean latency for a workflow looks fine at 120ms, but the 99th percentile is 2.4 seconds. That hidden entropy—the long tail—is where your users churn. The pattern: baseline the entire distribution, not just the center. Use the p50, p95, p99, and p99.9 as separate signals, each with its own sliding window. A shift in p50 means something systemic changed—code push, database load, cache miss. A shift in p99.9 alone usually means a spiky client or a GC pause. Different entropy, different fix. The trade-off: maintaining four baselines per workflow multiplies alert noise if you don't wire them to different severity levels. P50 drift gets a warning. P99.9 drift gets a page. Most teams reverse the polarity and burn out. A rhetorical question worth sitting with: if your single-number baseline never fires, how many slow users are you ignoring?

One concrete fix I have used: plot the ratio of p99 to p50 over time. That ratio stays stable for healthy systems—usually between 3x and 8x. When it jumps above 10x, something is injecting entropy into only the tail. You can baseline that ratio directly: a single metric that captures distribution shape without storing four time series. Not perfect for every case—spiky workflows with bimodal latency patterns will fool it—but it surfaces hidden entropy faster than any average-based approach I have seen. Start there, then layer in individual percentiles where the ratio lies.

Anti-Patterns and Why Teams Revert

Blanket baselines across heterogeneous workloads

The most seductive mistake: one baseline to rule them all. A team measures latency p99 across two microservices — both serve data, both run on similar infra — so they pin a single threshold. Six weeks later, Service A is fine; Service B is silently shedding traffic at 3 AM.

Why does this stick? Because it looks clean on a dashboard. The catch is that hidden entropy — different request shapes, different caching layers, different garbage collection cycles — accumulates unevenly. One workload breathes; the other chokes. I have watched engineers spend two sprints arguing that "the baseline must be wrong" before admitting the workloads were never comparable. The fix is boring but necessary: split the baseline first, merge the dashboards second.

Manual weekly recalibration

A team sets a Monday-morning calendar reminder. Update the baseline. Move the sliders. Deploy. That sounds fine until the Wednesday before a holiday, when nobody remembers, or the on-call engineer recalibrates against a three-hour window that happened to be a traffic lull. Wrong order. Not yet. That hurts.

The organizational reason this persists is trust — or rather, the lack of it. Automated drift detection feels like a black box; manual recalibration feels like control. But manual creates a hidden tax: every week the person doing it has different context, different stress, different fatigue. The result is a baseline that oscillates more than the signal it's supposed to anchor. Most teams skip this part: you can't out-discipline entropy with calendar invites.

You can't out-discipline entropy with calendar invites. The entropy owns the calendar.

— conversation with a site-reliability lead, after their third Monday-morning incident

Ignoring entropy until it causes an outage

The third anti-pattern is the most common and the most expensive. A shared baseline sits unchanged for months. Workload B adds a new endpoint that returns larger payloads — no alert because the aggregate p50 still looks clean. Then Workload A, which never changed, starts timing out because the shared baseline's alerting budget was silently consumed by B's new behavior. The seam blows out.

Flag this for data: shortcuts cost a day.

Why do teams revert to this? Because fixing it before the outage feels like premature optimization. The pressure to ship features outweighs the invisible cost of accumulated drift. One concrete anecdote: we fixed this by adding a weekly diff check — not a recalibration, just a flag if entropy between the two workloads changed by more than 10%. That single check caught three near-misses in two months. The anti-pattern persists because "it worked last quarter" is easier than "let's prove it still works today." Pick the second option. Your pager will thank you.

Maintenance, Drift, and Long-Term Costs

Dashboard bloat from splitting baselines

The first sign of trouble is almost always a proliferation of dashboards. One team I worked with started with a single shared baseline for two workflows running on the same infrastructure. After three months, they had 11 dashboards, each containing near-duplicate panels with adjusted thresholds. The original baseline wasn't wrong—it just couldn't explain why one workflow degraded every Tuesday at 2pm while the other stayed flat. So they split. And split again. Each split added a new set of time-series queries, more alerts, more annotations. Dashboard bloat isn't just clutter—it's a tax on every future incident response. You spend the first ten minutes of any outage figuring out which dashboard actually applies to the broken service. The shared baseline had hidden the divergence, but splitting it created a maintenance surface area nobody planned for.

Alert fatigue from too many dynamic baselines

Dynamic baselines sound like the fix. Let each workflow compute its own seasonal decomposition, its own residual thresholds, right? Wrong order. The hidden entropy doesn't announce itself until you have 150 alert rules firing at different cadences—some hourly, some rolling weekly windows, one using a custom ARIMA that recalculates every four hours. What breaks first is human attention. I have seen on-call engineers mute entire dashboards because the dynamic baseline for workflow B kept triggering on Tuesday's batch job while workflow A's baseline stayed silent. They set a global override. Now both workflows miss real degradation. The catch: dynamic baselines reduce false positives in isolation but increase false negatives in aggregate because nobody can hold all the behavioral quirks in their head. You trade one flavor of noise for another.

'The drift doesn't happen on the chart. It happens in the calendar—quarterly reviews, team rotations, forgotten threshold adjustments.'

— site reliability lead, after reverting three months of baseline customizations

Cost of analyzing false negatives from shared baselines

False negatives are the expensive kind of silence. When a shared baseline masks a degradation in one workflow because the other workflow's healthy variance drowns it out, you don't notice until the business impact shows up—slower page loads, higher error rates, a support ticket from the biggest customer. Then you spend four hours reconstructing what happened: pulling raw metrics, re-running analysis with isolated baselines, comparing the two workflows side-by-side. That hour of detective work is the real cost. Not the compute, not the tool licensing—the engineering time spent proving the baseline was wrong instead of fixing the actual problem. Most teams skip this step. They see the alert didn't fire and move on, assuming everything was fine. The hidden entropy gets a free pass until the next quarterly review surfaces a correlation they should have caught months earlier.

The fix? Schedule explicit baseline audits every six weeks—not for the thresholds, but for the divergence itself. Compare the two workflows' actual distributions over a sliding 30-day window. If the gap widens beyond 2 standard deviations on any percentile, split the baseline before someone has to file a postmortem. That sounds fine until the calendar slips and nobody owns the audit. Then you're back to drift, back to bloat, back to the same expensive silence. The long-term cost isn't the tooling—it's the discipline to maintain a shared abstraction that was already cracking under the surface.

When NOT to Use This Approach

When workflows have radically different performance profiles

Some baselines deserve to die alone. If Workflow A processes 10,000 events per second with 99th percentile latency under 50ms, and Workflow B churns through 12 events per hour with occasional 12-second pauses—sharing a baseline is cargo-cult efficiency. The entropy isn't hidden then. It's screaming at you. I once watched a team merge baselines for a payment gateway and a background report generator. The gateway's p99 looked fine until the report job kicked off and dragged the shared floor up by 40ms. False alerts everywhere. The fix? Split them and never look back.

What counts as "radically different"? Look at amplitude ratios. If one workflow's peak-to-median spread is 3x and the other's is 30x, you're measuring two different animals. The shared baseline will optimize for the loudest signal—usually the noisy one—and bury the quiet workflow under a blanket of irrelevant thresholds. Worth flagging: this mistake is most common when teams inherit a monitoring setup designed for a single service, then bolt a second workflow on top without re-evaluating the baseline logic. The cost is one long week of pager fatigue and a ticket titled "disable alert for endpoint X."

'A baseline shared across mismatched profiles is not a baseline. It's a lie wrapped in a dashboard.'

— site reliability engineer, after untangling a shared baseline for a cron job and a real-time API

When you lack instrumentation to segment baselines

You can't carve a baseline you can't see. If your telemetry only gives you a single aggregated metric for both workflows—say, a combined request count or a lumped error rate—then shared baselines aren't a choice. They're a trap you're already in. The obvious alternative is to instrument each workflow with a distinct label or metadata tag before you even think about baseline logic. But teams often skip this step because it feels like "overhead" during a feature push. They tell themselves: we'll add the tags later. Later never comes.

The pattern I see most often: an ops person notices weird behavior, adds a shared baseline as a band-aid, and then the band-aid calcifies into production monitoring. Two quarters later, nobody remembers why the baseline exists or what it's supposed to protect. The fix is boring but surgical—instrument first, baseline second. If your current stack can't tag workflows separately, consider a lightweight sidecar or a simple wrapper that injects a workflow_id header. That sounds fine until your CEO asks why alerts went dark for 45 minutes while a silent failure bled through a shared error-rate baseline. Then the instrumentation budget appears.

Reality check: name the quality owner or stop.

When your team is too small to maintain separate baselines

This one hurts because it's a resource constraint, not a technical one. I've been there: a team of two engineers, 15 microservices, 37 cron jobs, and a Slack channel that smells like burnout. Separate baselines per workflow sound great on a whitepaper, but who will tune them weekly? Who will review drift reports when both people are on call for the same alerts? In that scenario, a shared baseline—even a leaky one—beats no baseline at all. The catch is that you must accept the noise. Accept the false positives. Accept that hidden entropy will sometimes slip past.

The pragmatic alternative: start with one coarse baseline for all workflows, then layer a single "critical path" override for the one workflow that can't tolerate a shared floor. Pick the workflow that pays the bills or keeps the database alive. Everything else gets the shared bucket. This isn't pretty, but it's honest. You trade precision for survivability. Later, when you hire a third engineer or get a monitoring budget, you carve out the next workflow. Most teams skip this: they try to build the perfect segmentation on day one, fail, and abandon baselines entirely. A partial win beats a full retreat.

FAQ: Hidden Entropy and Shared Baselines

How can I detect hidden entropy without instrumenting every endpoint?

Instrument everything is the textbook answer — and the textbook is lying to you. In practice you can spot entropy asymmetry by watching one cheap signal: the timing shape of error recovery. I have seen two workflows share a baseline, identical failure rates on paper, but one took 400ms to recover and the other took 12 seconds. The slow one was silently accumulating state garbage. That's hidden entropy — not a spike, not a crash, just a slow leak of predictability. Stop watching means. Watch the tail of recovery latencies. If the 95th percentile drifts apart from the 50th in one workflow but not the other, you have found the asymmetry without instrumenting a single additional call.

What's the minimum viable segmentation for baselines?

Most teams over-split. They carve baselines by region, by deployment tag, by customer tier, by phase of the moon. The catch is — more segments mean more baselines to maintain, and maintenance is where entropy eats teams alive. The minimum viable segmentation is two groups: the workflow that changes often and the workflow that stays still. That's it. I fixed a production issue once where a shared baseline hid a growing memory leak in a batch processor, while the real-time path stayed clean. The team had ten segments. They needed exactly two. Wrong order: start with too many segments, and you will never notice when one starts sneaking entropy because you will blame the segment boundaries instead.

“A shared baseline isn’t a contract. It’s a hypothesis that two systems age at the same rate.”

— senior engineer who learned this the hard way, after a three-day incident

Should I ever merge baselines back together?

Yes — but only when you prove entropy symmetry first. That sounds like extra work, and it's. Worth flagging: merging prematurely is the fastest way to blind a team to regressions. Run a shadow comparison for two full release cycles. Check that the variance of the two workflows overlaps within a tolerance you define before looking at the data — picking the threshold after seeing numbers is cheating yourself. What usually breaks first is that one workflow gets a dependency upgrade nobody recorded, and suddenly its error distribution shifts. If you merged, that shift looks like normal baseline noise. You lose a day. Then another. That hurts. The only safe merge is one you can revert in under ten minutes, with data proving both sides still breathe the same air.

The next experiment? Pick one shared baseline pair in your dashboards — the one you trust most — and split it for one week. Run the math on recovery tails. You might find entropy you never knew was there.

Summary + Next Experiments

Key takeaways

Shared baselines break when hidden entropy lives in just one workflow. That entropy isn't always noise—sometimes it's a slow memory leak, a caching layer that expires differently, or a downstream API that degrades only during certain hours. The baseline looks clean because the other workflow masks the problem. I have seen teams chase phantom thresholds for weeks, only to discover that one service was retrying silently while the other wasn't. The real lesson: a baseline is an average of behavior, not a guarantee of health.

Most teams skip this—they assume two workflows with similar p50 latency or error rates share the same risk profile. Wrong order. The hidden entropy lives in the tail, in the request patterns, in the resource contention that only one path triggers. You can't fix what you haven't split. That hurts because splitting baselines costs time, storage, and alert configuration overhead. But the alternative is worse: missed incidents or, just as bad, alert fatigue from a baseline that tries to serve two masters.

A baseline that fits two workflows equally fits neither well enough to trust during an incident.

— Senior SRE, after untangling three false alarms in one shift

Immediate steps to audit your shared baselines

Start with the p99. Not the average—the p99. Pull the same signal (latency, error rate, request volume) for each workflow separately, even if you currently aggregate them. Plot them on the same timeline. If one workflow's p99 jitters while the other stays flat, you have found the entropy. Next, check the gap between p50 and p99 for each workflow separately. A gap that's 2× wider on one side than the other is a red flag—something is adding variance that the aggregate baseline is absorbing. The catch is that this audit takes about ninety minutes the first time. After that, it becomes a monthly check you automate.

What usually breaks first is the alerting rule. When the shared baseline drifts because one workflow's traffic pattern changes (say, a batch job that now runs hourly instead of daily), the other workflow inherits a threshold that no longer fits. The fix is brutal but fast: create a second alert rule using the same signal but scoped to the specific workflow. Run both in parallel for one week. Compare false-positive rates. If the split version produces fewer misfires, you keep it. That's the experiment you run tomorrow morning—not next sprint.

Experiment: split one baseline and compare alert accuracy

Pick the most annoying false-alarm-prone system you have—the one that pages at 2 AM and turns out to be nothing. Extract its signal into its own baseline. Don't touch the other workflow's configuration yet. Run both the original shared baseline and the new split baseline for two weeks. Track three metrics: number of alerts, mean time to acknowledge, and how many alerts were dismissed as noise. The bet is that the split baseline either reduces alert volume or improves acknowledge speed—or both. If it does neither, you can merge back without shame.

The tricky bit is that some teams see worse accuracy at first. That happens when the split baseline has too little data—only two days of history, for example. The baseline needs at least seven days of the workflow's own rhythm before it stabilizes. Wait that out. If after two weeks the split baseline still triggers more noise than the shared version, then maybe your two workflows are actually the same under the hood. But I have not seen that happen yet—hidden entropy always reveals itself eventually. The question is whether you catch it before the next on-call shift burns out.