Story 01 / Progressive delivery
Your Deployment Succeeded. Your Customers Weren't So Lucky.
Progressive delivery, canary releases and release safety. Explore customer impact, compare release strategies and decide what evidence should justify the next step.
The deployment worked. Checkout didn’t.
A customer presses Pay, waits a moment and lands on a page that tells them nothing useful. There’s no order number and no confirmation. They could try again, but they might have already paid. From the shop’s side, the release looks uneventful: the new application is running, the servers aren’t busy and no deployment check has failed.
Imagine this happening at Fold & Field, an online shop releasing a change to its checkout. The shop and its figures are fictional, but the gap between those two views of a release is the problem we need to understand.
Fold & Field / A fictional release
The deployment pipeline has replaced the application with a new version. All four copies, usually called replicas, have started and passed their readiness checks, so the traffic router can send requests to them. That is useful evidence. A version that cannot start should never be put in front of customers. It is not yet evidence that the checkout works.
A readiness check answers the question its developers gave it. It might confirm that the process is responding or that a required connection is available. Unless it exercises the relevant checkout behaviour, it cannot tell us whether a customer who pays will receive confirmation. Adding more checks helps only when they observe something the existing checks miss.
In the new version, one in five checkout attempts follows a faulty path. The application records the order, the payment is recorded and a fulfilment message is accepted. The code that builds the confirmation page then fails, and a fallback returns an unhelpful page with HTTP 200, the status normally used for a successful web response. The server has replied successfully in the narrow sense measured by an HTTP status counter. The customer still doesn’t know whether the purchase worked.
This is why deployment success, application health and customer success can disagree without any of the individual measurements being broken. Each describes a different part of the system. The deployment job checked that the version was installed. Readiness checked that an application copy could receive traffic. Neither followed the customer from payment to a visible confirmation.
The distinction matters because the consequences continue beyond the failed page. A retry might create more work or another payment unless the service recognises it as the same purchase. Support may need to find an order that the customer cannot identify. Sending traffic back to the previous version can prevent more attempts from hitting the defect, but it doesn’t remove the records already written.
Suppose the shop receives 1,000 checkout attempts a minute. If everyone reaches this version for five minutes, a 20% failure rate produces 1,000 expected missing confirmations. That counts failed attempts, not unique customers or lost orders. The immediate release problem is to discover the failure and limit further exposure while there is still a practical recovery path.
We could respond by expanding the test suite, and a test that catches this particular path would be valuable. We should also ask what happens when the next defect escapes it. A release process needs a way to learn from genuine production use without treating every customer as the first test. That means separating two decisions which pipelines often make together: whether a version is ready to run and whether more people should receive its behaviour.
Two versions, one checkout
then a ready copy
Follow a request
Example paths, not a measured traffic sample.
When Fold & Field deploys its next checkout update, it doesn't have to replace the old version straight away. It can keep v1 serving customers while v2 starts alongside it, giving the team control over when customers begin using the new version.
Deployment makes v2 available to run, while release lets customer requests reach its new checkout behaviour. A pipeline often does both together, but the team can keep those decisions separate.
For this to work, the router needs a way to reach each version separately. Imagine four application copies running v1 and two running v2. A destination called checkout-v1 contains the addresses of ready v1 copies. Another, checkout-v2, contains only v2 copies. Service discovery can keep these lists up to date as copies start and stop, using version labels to identify which copies belong in each list. What matters is which application copies each destination selects, not just that the destinations have different names.
With those destinations in place, both versions can run on production machines while normal checkout traffic still goes to checkout-v1. Starting two v2 copies gives the router somewhere else to send a request. It doesn't change where the public route currently sends traffic.
The customer still uses the shop's usual checkout address. Behind that address, an application router, such as a proxy or load balancer, follows a rule to choose a version destination. It then selects a ready application copy within that destination and forwards the request. Choosing the version and choosing a copy of that version are separate steps.
A weighted rule could give v1 a weight of 90 and v2 a weight of 10. If the router chooses a version independently for each request, those weights produce roughly a 90/10 split over enough traffic. That doesn't mean exactly one request in every group of ten reaches v2. A short sequence could contain several v2 requests or none at all. The illustration shows possible request paths, not measured traffic or a guaranteed split.
The routing rule also explains why replica counts don't necessarily tell us how traffic is distributed. With four v1 copies and two v2 copies, a third of the copies run v2, but its destination still has a routing weight of ten per cent. Adding another v2 copy increases its application capacity without having to change that weight.
If both versions instead share one destination without version aware weighting, changing their replica counts may change the traffic split. How closely it follows those counts depends on the load balancing policy, connection behaviour and the work each copy receives. Separate version routing lets the team adjust the traffic split independently of the number of copies. Each pool still needs enough capacity to serve the traffic sent to it.
Those weights describe requests, but a customer makes several requests during checkout. If the router chooses a version independently each time, the same customer could load a basket through v1, submit payment through v2 and ask for confirmation through v1. Sending ten per cent of requests to v2 doesn't tell us what proportion of customers have used it. Customers who make more requests have more opportunities to reach both versions.
That mixing may be acceptable when both versions understand the same session and data formats. If the journey needs to stay on one version, a session cookie can record the assignment, or a routing rule can use a stable account identifier to assign the customer to a consistent group. Requests carrying that identity then keep reaching the same version while the assignment holds.
The assignment needs to happen where the router chooses the version. A sticky session that keeps requests on one application copy within a pool doesn't necessarily keep later requests out of the other version's pool. Consistent routing also doesn't make session data available automatically: the application still needs to find and understand the customer's session. Even with consistent groups, ten per cent of accounts might generate far more or less than ten per cent of requests, because people use the shop differently.
Keeping the application versions separate doesn't separate everything they use. Both pools may still write to the same order database, call the same payment provider and place work on the same queue. A request served by v2 can affect something that v1 uses too. Separate destinations let us choose which code handles a request, but they don't create two independent systems.
Both versions can now run in production without every customer immediately receiving the change. We can choose how requests reach each version and adjust that split separately from application capacity. Now that we can control exposure, how should we decide which release strategy to use?
Implementation references: Envoy’s weighted traffic routing and Argo’s separate version destinations illustrate the mechanism. Argo’s replica based approximation shows why replica counts and explicit routing weights are different controls.
Ready for customers?
Fold & Field's next checkout version has passed its tests. The existing version is working, and the team can run the replacement alongside it. The remaining decision is how to put it in front of customers without giving up the ability to respond if something goes wrong.
That decision involves more than the pace of replacing application copies. Preparation gets the replacement running. Exposure brings it into contact with real production work. Validation tells us what that work reveals, and recovery gives us a way to respond. A release approach can make one of these easier while leaving the others largely untouched.
Fold & Field / One checkout, different releases
4 running copies
1 running copy
Full v1 capacity is still running. Returning traffic also requires compatible state and a working route.
Illustrative snapshots, not timed steps or an automatic promotion rule. Copies shown are ready and equally sized; assigned traffic is assumed to fit. Counts exclude terminating copies. Weights describe requests, not people.
Replacing the application a few copies at a time
A rolling deployment replaces old copies gradually rather than stopping the whole application at once. For a checkout running four copies, the team might start one new copy, wait for it to become available, then remove an old one. The process repeats until all four run v2. Once a new copy is ready and joins the serving pool, customers can reach it while replacement continues.
Kubernetes makes two of the capacity choices explicit. maxSurge allows extra Pods above the desired count during replacement. maxUnavailable limits how many can be unavailable. With four desired copies, maxSurge: 1 and maxUnavailable: 0 allow an extra copy while keeping at least four available as the rollout progresses. This needs room to start the extra copy. Copies still terminating can consume resources beyond that working count.
Those controls manage replacement and availability. A normal rolling update doesn't inherently stop to evaluate whether customers can complete checkout before replacing the next copies. Readiness and availability can allow it to continue while a defect like the missing confirmation escapes them. The arrangement is useful, but business validation needs a separate mechanism.
Kubernetes Deployments documents rolling replacement, availability limits and terminating Pods.
Preparing the replacement before the cutover
A blue/green arrangement keeps the existing environment serving while the team prepares a complete replacement. Fold & Field could start four v2 copies behind a preview address, check their configuration and exercise checkout before moving the production route. Here, a complete replacement means application capacity. It doesn't imply a second database or payment system.
Keeping v1 running after the cutover can give the team a useful return path without waiting for old application copies to start again. That costs extra capacity and operational work: the team must maintain both environments, validate the replacement and decide when the old one can go. Returning traffic still depends on the old version being usable with the current data and dependencies.
The preview removes some uncertainty, not all of it. Tests may miss a payment path, a configuration difference or behaviour that appears only under the production workload. A single intended cutover can expose the whole checkout to such a defect. Routing changes also take time to propagate, and requests already in progress don't disappear when the route changes. Retaining the old environment is a recovery option, not proof that the cutover will succeed.
Argo's blue/green implementation illustrates separate preview and active destinations and delayed retirement during routing propagation.
Learning from limited production exposure
A canary release lets a limited share of real production work reach v2 while the team evaluates its behaviour. The new application might begin as a small pool alongside v1, with enough capacity for its assigned traffic. If the release decision supports increasing exposure, the team can add new capacity and reduce the old pool as it proceeds.
This accepts some customer exposure in exchange for evidence that tests couldn't provide beforehand. If the confirmation defect appears, limiting the affected requests may prevent a much larger incident. But the first affected customer still has a real problem. A small share is not automatically acceptable for payments, and a defect that damages shared state may affect work outside that share. Canarying also depends on recognising meaningful failures and acting on them. Merely sending fewer requests to v2 doesn't provide either ability.
Google's canarying guidance treats evaluation as part of the release strategy. The percentages in the illustration are examples, not safety thresholds.
Preparing fully, then exposing gradually
Blue/green and canarying answer different questions, so they can work together. The team can prepare all four replacement copies, validate them through the preview route, then send a limited share of production work to that environment. It retains the original application capacity while deciding whether to complete the cutover.
For a checkout that takes time to start or needs a ready return path, that combination may justify the extra application capacity. It also means operating two environments and controlled exposure, with all the associated routing, validation and recovery work. The full replacement doesn't remove the need to evaluate it, and gradual exposure doesn't remove the cost of keeping it ready. Before paying for both, the team needs to check what running both will do to the rest of production.
What starts before the first customer arrives
Keeping the application versions separate doesn't separate everything they use. Suppose each of the four v1 copies holds 20 database connections. The four replacement copies are also configured for pools of up to 20. Together, those pool limits allow a potential demand of 160 connections, even while the public route still sends every checkout request to v1.
A pool maximum isn't a count of open connections. Some applications open connections as work arrives. Others establish a minimum pool during startup or keep idle connections ready. If the checkout has a budget of 100 available database connections and both environments actually try to hold all 160, they can't all be admitted. The replacement could struggle to start or serve its first requests before any traffic cutover. Existing connections, retries and competition from other clients affect what happens next.
Fold & Field / Under the application layer
Public checkout → v1 v2: 0% HTTP trafficv14 existing copiesv24 replacement copiesLift the application layer
Each paper socket represents 5 admitted connections.
80 already held by v180 requested by v2100 admitted in total
160 connections requested against 100 available slots. 60 cannot be admitted. v2 still has no public HTTP traffic.
Illustrative connection budget, not a live database. Four v1 copies already hold 20 connections each. Four v2 copies each have a pool maximum of 20. Here v2 also tries to open all 20 at startup. Total configured maximum: 160. The 100 available slots allow for other clients and reserved connections already. No connection proxy, release of existing connections or retry behaviour is modelled.
The reveal assumes v1 already holds 80 connections and gives the checkout 100 usable slots after allowing for reserved connections and other clients. Real PostgreSQL installations enforce max_connections, with some slots reserved for privileged connections. The application's usable budget can therefore be lower than the server setting. A connection proxy can also let application connections share fewer database connections, so adding up application pool limits alone may overstate demand on the database. The team needs to verify its actual pooling and startup behaviour.
PostgreSQL connection settings describe the server limit and reserved slots. The four copy example and connection policies are illustrative.
Background work creates another route into production. If the new checkout starts a RabbitMQ consumer on the existing work queue, that consumer may receive messages without handling a single public HTTP request. In a typical queue with competing consumers, deliveries are distributed among eligible consumers, influenced by settings such as prefetch, consumer priority or single active consumer mode. Adding a consumer doesn't mean every message is automatically processed twice, but it can put real work through v2 earlier than the HTTP release plan intended.
Kafka uses a different model. Consumers in a subscribing consumer group receive assignments of topic partitions. Bringing v2 into the same group can trigger reassignment as membership changes. The amount of disruption depends on the group protocol and assignment configuration; it isn't always a pause across the whole group. This is not RabbitMQ's per message queue dispatch. In either case, the team must know whether starting the replacement also starts production processing.
See RabbitMQ consumers for queue delivery behaviour and Confluent's Kafka consumer documentation for groups and rebalance protocols.
The same check applies to scheduled jobs, cache warmers and startup calls to a payment provider. A second scheduler could run work unless ownership is coordinated. Both versions might consume the same downstream quota or write to the same cache. None of this means extra copies inevitably overload production. It means the release boundary must include the work and resources the application actually uses. Staging worker activation separately, limiting connection pools or using isolated test dependencies may matter more than the public traffic weight.
A release the checkout team can support
For Fold & Field, a prepared replacement with controlled production exposure is a reasonable starting choice. A failed checkout leaves customers uncertain about payments and orders, so learning before a full cutover has value. Retaining v1 capacity also gives the team an application return path. That recommendation depends on both versions being able to coexist, the team being able to recognise meaningful checkout failures and shared dependencies having room for the overlap. Background consumers need their own activation plan. If one operation could cause unacceptable harm, a smaller traffic share is not sufficient protection.
If a full second application environment adds no useful recovery capability, the team could use a smaller canary pool instead. For an application with simpler consequences, strong tests and a rehearsed redeployment, an ordinary rolling update may be easier to operate well. The choice should follow the cost of customer harm, the evidence available and the recovery the service needs. Additional release machinery has a maintenance cost and can fail too.
A release strategy can reduce particular risks without eliminating uncertainty or isolating the whole system. For this checkout, the prepared environment and controlled exposure give the team a way to learn and respond. Now that we've chosen how to introduce the release, what evidence would convince us that it is safe to continue?
How do we know it's working?
Fold & Field has prepared the replacement checkout and given it ten per cent of production requests. The original application is still available. Both versions are running normally, readiness checks pass and the HTTP success rate looks healthy. On that evidence alone, there seems little reason to hold the release back.
The confirmation defect hasn't gone away. On the faulty path, the order is saved, payment is recorded and a fulfilment message is accepted. The confirmation page then fails, but the fallback still returns HTTP 200. Each measurement describes something that really happened. The application replied, the payment was recorded and the process stayed alive. None of those facts establishes that the customer received a usable confirmation.
Technical health is still useful evidence. Restarts, saturation and slow requests can expose problems or help explain them. The mistake is asking those measurements to answer a question they don't cover. For this release, we need to know whether customers can finish checkout and understand the result.
Google's monitoring chapter distinguishes internal measurements from externally visible behaviour. Both matter, but they establish different things.
Following the purchase far enough
A service level indicator, or SLI, is a measurement of service behaviour that matters to its users. For this checkout, one useful indicator would be the proportion of eligible attempts that produce a visible, usable confirmation within an agreed time. The numerator counts those successful outcomes. The denominator counts the eligible attempts that had the opportunity to complete, including the ones that failed.
That sounds straightforward until we decide where an attempt begins. A page view isn't a purchase attempt, and several network retries might belong to one submission. The team could record an attempt when the service accepts a valid purchase submission, with a stable identifier carried through its processing. It also needs explicit treatment of legitimate payment declines and abandoned journeys. A failed attempt must not disappear from the population simply because it never produced an order row.
The timing matters too. Join the outcome to the attempt that started it, rather than divide confirmations received this minute by submissions received this minute. Some of those confirmations may belong to earlier work, while recent submissions haven't had time to finish. Group attempts by when they started, allow the agreed completion time and account for late telemetry. Keep pending outcomes separate until they can be classified. Otherwise, ordinary processing delay can look like a regression, or missing events can quietly vanish from the calculation.
An order record and a payment event can establish progress through checkout. To observe confirmation more directly, the page could emit an event after rendering the expected order reference and confirmation content. That is better evidence than a successful HTTP status, but it still isn't a perfect account of what the person saw. Browser telemetry can be blocked or lost, and someone may close the tab before the event is sent.
Confidence comes from connecting signals with different blind spots. A backend event can record a confirmation builder error. A browser event can report that confirmation content rendered. A synthetic checkout can test a known journey, though it cannot represent every real customer. Missing browser evidence might mean a broken page or a broken measurement path. Record that uncertainty rather than invent a success or treat every absent beacon as a proven application failure.
A correlation identifier lets the team connect the submission, order, payment and confirmation events across service boundaries. Trace context can link the calls involved; a durable attempt identifier can also join later events. Record which application version performed the relevant operation when it happens. Looking up the current release label afterwards can assign old work to the wrong version. If a journey crosses versions, retain those operation level labels rather than pretend one version did everything. Keep detailed identifiers in suitable event records and aggregate outcome counts by release and relevant customer characteristics.
Google's SLI guidance separates the outcome we want to measure from its practical implementation. OpenTelemetry's context propagation documentation explains how related signals can be connected across services.
Fold & Field / The first five minutes
v1 90% · v2 10%5,000 expected attempts
The replacement is running.
8 / 8copies ready across both versions
32%fleet CPU use
0restarts
These fictional readings establish that the copies can receive work. They haven't followed a customer from payment to confirmation.
A successful response can contain a failed checkout.
5,000 / 5,000expected final responses return HTTP 200
240 msresponse p95 on each version
- Order saved
- Payment recorded
- Message accepted
- Confirmation builder fails
- Fallback returns HTTP 200
This is one attempt on the established faulty path. Instrumenting the confirmation builder could expose its failure even while the HTTP success counter stays green.
Payment recorded is not the end of the journey.
The aggregate includes 4,500 stable attempts and 500 canary attempts. The same 100 missing confirmations are concentrated within v2.
The same total, a different view of the release.
4,500 confirmations / 4,500 attempts
Response p95: 240 ms400 confirmations / 500 attempts
Response p95: 240 msWhole shop: 98% confirmation success. Different denominators explain the diluted view.
A difference is a reason to investigate the release, not proof of its cause. The comparison also needs comparable work and an absolute view of customer health.
Inspect gaps and shared effects
The established confirmation defect remains unchanged. These options change what the team can observe.
Separating the release from the rest of production
The whole shop result can make this defect look smaller than it is. At 1,000 attempts a minute, a ten per cent canary share gives v2 roughly 100 attempts a minute. Over five minutes, that is an expected 500 attempts. With its twenty per cent defect rate, 100 would lack confirmation. The other 4,500 attempts use v1 and, in this simplified example, all receive confirmation.
Together, that gives 4,900 confirmations from 5,000 attempts: 98% across the shop. Separately, v1 has 4,500 from 4,500, while v2 has 400 from 500. The new version's 80% result is much harder to overlook. These are expected outcomes for attempts admitted during the five minute cohort, read after their completion allowance, not measured production counts. For the arithmetic, each attempt is one routed checkout operation attributed to its handler version. The request weight is not a count of unique customers.
The stable population gives us a useful comparison because it is doing production work during the same period. It isn't automatically a fair comparison. If v2 mostly receives mobile purchases, one geography or a difficult payment provider, the difference could reflect that mix. Account type and request behaviour matter too. Compare relevant groups and check that the traffic exercises the behaviour we intend to release. A large sample of the wrong work doesn't become representative simply by getting larger.
A shared dependency can complicate the picture in the other direction. If the payment provider slows down, both versions may get slower together. The illustration adds that shared effect to the response time readings while leaving the confirmation defect unchanged. In production, a shared failure could damage both populations' outcomes as well. A small difference between versions would then be poor reassurance about the service customers are receiving.
Keep the relative comparison alongside the absolute customer outcome. A difference associated with v2 is evidence to investigate, not proof that its code caused the problem. Conversely, a common deterioration doesn't establish that the release is innocent: v2 could have increased pressure on a dependency shared with v1. Understanding the cause and recognising that customers are receiving bad service are related questions, but they are not the same question.
Google's canarying guidance discusses comparison populations, representative work and shared dependencies.
What a quiet release has actually told us
Even a well chosen signal needs opportunities to reveal a problem. A quiet canary might receive only a few attempts. An expected failure rate does not require every short sequence to contain a failure, so a clean sequence can coexist with a defective release. The relevant question is which behaviour has been exercised, how often and under which conditions.
Fold & Field / Time leaves different kinds of evidence
One minute
100 expected v2 attempts20 expected missing confirmations.
A short window has exercised only the work that arrived.Five minutes
500 expected v2 attempts100 expected missing confirmations.
More attempts can reveal more about the journeys actually exercised.Work beyond the window
Accepted is not completedThe fulfilment message exists. Its eventual processing is not measured by these counts.
A later callback or job result needs its own observation.
Volume does not establish coverage. An unvisited payment route or device remains untested, however busy the rest of checkout becomes.
A quiet minute can contain no failures
At 50 total attempts a minute and the same ten per cent split, v2 would receive five attempts on average. A twenty per cent defect rate gives one expected failure, but an expectation is not a quota. These five receipts show one possible sequence in which all attempts avoid the faulty path. They do not show that the defect has gone away.
If the canary excludes the affected journey altogether, even a much larger clean result could miss the defect. This is a different limitation from simply having few attempts.
Volume and time answer different questions. More representative attempts can give us more evidence about an immediate confirmation defect. They may still tell us nothing about an uncommon payment route that hasn't been used. Repeated attempts from the same account or device can also add less variety than the raw count suggests. If the cohort excludes the failure condition, waiting longer on the same narrow population may add little useful evidence.
Some behaviour cannot be evaluated until later work happens. The accepted fulfilment message in our checkout is evidence that work entered the system, not that it was completed correctly. A background job or external callback may arrive after the initial release window. Follow that work to its relevant outcome and preserve the version and attempt context needed to interpret it. A busy few minutes cannot substitute for observing a delayed process.
There is no universal traffic percentage or observation period that settles all of this. The team needs enough representative work to exercise the risky paths, enough time for relevant delayed behaviour and a measurement path it can trust. A missing event, an idle canary or an unvisited journey leaves an evidence gap. More precise looking numbers don't fill it.
An inconclusive result is useful information about what we don't yet know. Release tooling can represent that distinction too: Argo's analysis model includes an inconclusive outcome rather than forcing every result into success or failure. The engineering decision still depends on what is missing and what harm could occur while we learn more.
For Fold & Field, HTTP success is insufficient, the aggregate hides the concentration of missing confirmations and the version comparison needs comparable work. We can now describe the evidence we need and the limits of what it establishes. We have evidence from the release. Who decides whether it is good enough to continue, pause or roll back?
When should we stop a release?
Fold & Field has the evidence it needed. In our fictional five minute cohort, v1 produced 4,500 confirmations from 4,500 attempts. v2 produced 400 from 500. The missing confirmations are a serious regression, even though the instances remain healthy and the fallback pages return HTTP 200. This is not an inconclusive result that needs a more reassuring interpretation.
The next action should be to stop increasing exposure and withdraw the defective version from new production traffic, assuming v1 can still handle the load. Collecting another set of measurements while leaving the same defect in front of customers would allow more harm. The release process needs to turn this evidence into an operational response.
A release quality gate makes that connection explicit. It evaluates observations against predefined conditions and determines whether the next stage is permitted. Those conditions should be agreed before the release, when the team can consider customer harm, recovery requirements and the consequences of continuing. A successful deployment job can be one prerequisite. It cannot override a confirmation result that shows the service failing its customers.
The policy needs both acceptance conditions and stop conditions. It should say which evidence must be available, what behaviour would be unacceptable and what to do when the result cannot be evaluated. It also needs a response for each outcome. A gate that reports failure while the release continues is a warning label, not an effective safeguard.
Stopping progression is not stopping exposure
A pause can prevent the controller from advancing beyond the current stage while leaving the current route unchanged. At ten per cent, v2 would still receive new checkout work. With the established demand and defect rate, each further minute would bring twenty expected missing confirmations. Pausing progression alone is therefore inadequate for this incident. The version already serving customers is the one causing the harm.
Aborting this release should stop its progression and initiate withdrawal from new v2 traffic. That is more than changing the release status. Something must update the routing configuration, the components enforcing that route must receive the change, and the team must check that newly routed work is reaching the intended destination. A requested weight of zero and evidence that the effective route has changed are different facts.
The stable destination also has to be usable. If its capacity was reduced during the rollout, redirecting the entire checkout there could create another incident. Fold & Field retained its original application environment, but the response still depends on verified capacity and compatible behaviour. If that assumption fails, the withdrawal remains necessary while an engineer establishes an appropriate containment path. An unexecuted abort command is not a safe hold.
An abort also doesn't mean the same thing in every delivery system. A controller might change traffic weights, replace replicas or require an external routing action. Its permissions and integration can fail too. Requests already running may continue, and changing where new work goes does not erase completed operations. The response has to be understood and checked against the system actually in use.
Argo's canary documentation shows why retained stable capacity matters when traffic returns. Its behaviour depends on the routing and scaling configuration, not just the name of the action.
Fold & Field / Evidence → policy → response
Current release share: 10%Established incident
The customer regression is clear.
- Stable v1
- 4,500 / 4,500 confirmations
- Canary v2
- 400 / 500 confirmations
Policy response: Withdraw new v2 traffic
Known customer harm blocks promotion and requires withdrawal from new v2 traffic. Pausing progression alone leaves the defective version serving.
Evidence packet details
The five minute cohort has 4,500 confirmations from 4,500 v1 attempts, and 400 from 500 v2 attempts. The 100 missing confirmations are expected outcomes under the established fictional model. HTTP responses and application health still look successful.
Hypothetical alternative
No observed failure, but the required journeys are missing.
- Stable v1
- No failure reported in the small cohort
- Canary v2
- Coverage and observation incomplete
Policy response: Hold and investigate
The evidence does not justify promotion. Hold progression for a bounded review, assign an owner and decide whether continued exposure is acceptable. A clean small sample is not permission to expand.
Evidence packet details
A different release has handled only a handful of eligible attempts. None failed, but important journeys have not been exercised and the agreed observation requirements are not met. This does not replace the established incident.
Hypothetical alternative
The required result cannot be evaluated.
- Stable v1
- Confirmation outcome unknown
- Canary v2
- Confirmation outcome unknown
Policy response: Hold and investigate
Block promotion and escalate the evidence gap. This example holds the current share only during a bounded investigation. If the risk cannot be observed acceptably, the owner must arrange withdrawal or other containment.
Evidence packet details
In this separate example the measurement provider is unavailable, the query is invalid or the required customer events are missing. No reliable confirmation rate can be established. An unavailable result is not a successful result.
Hypothetical alternative
The service has a wider problem.
- Stable v1
- Customer outcomes deteriorating
- Canary v2
- Customer outcomes deteriorating
Policy response: Hold and escalate incident
Stop expansion and escalate a service incident. The illustration does not choose a traffic remedy from the relative comparison alone. Current exposure is an interim state, not evidence that either population is safe.
Evidence packet details
In this separate example, stable and canary customer outcomes both violate their agreed conditions. A shared dependency may be failing, or the release may have increased pressure on it. Similar results do not identify the cause.
Hypothetical repaired candidate
The agreed evidence permits one more stage.
- Stable v1
- Agreed customer conditions met
- Canary v2
- Coverage, time and acceptance conditions met
Policy response: Permit one next stage
Permit the configured next stage while continuing to observe it. This example uses a 25% next share, not a recommended safe percentage. Passing this gate does not approve every later stage.
Evidence packet details
A separate, repaired candidate has valid observations, the required representative journeys and delayed checks. Its agreed observation requirements and customer acceptance conditions are met, with no known stop condition. These are stipulated inputs, not inferred from an invented count or duration.
Policy target for v2 0%
Routing action No command issued
The existing cohort still contains 100 expected missing confirmations. A routing change does not alter those records.
Progression blocked
Progression is held, but new traffic still reaches v2 at 10%. At the established demand and defect rate, that means 20 expected missing confirmations for each further minute. Holding alone is not containment.
The policy behind these examples
- Known material customer harm from the candidate blocks promotion and requires withdrawal from new traffic, with a verified destination for that traffic.
- Incomplete or unavailable evidence blocks promotion and requires an owner and a bounded review. Continuing exposure needs a separate risk judgement; a hold is not an indefinite safe state.
- Both populations deteriorating blocks expansion and triggers service incident investigation. The comparison alone does not choose the remedy.
- Valid evidence meeting the agreed coverage, observation and customer conditions permits one configured next stage. Monitoring and stop conditions continue.
The example policy does not derive acceptance from a universal request count, duration or failure threshold. The repaired candidate's satisfied conditions are stipulated inputs. They must be defined and verified for a real service.
When a hold needs an owner
The confirmation defect gives us a clear reason to withdraw. Other releases can leave a genuine evidence gap. A canary that has handled only a handful of attempts may have no reported failures while missing the important journeys altogether. That does not meet the conditions for promotion. Holding progression can give the team time to gather the missing evidence, but the hold needs a purpose, an owner and a limit on how long that uncertainty will be tolerated.
The same applies when a measurement provider stops responding, a query is invalid or customer events stop arriving. None of those outcomes is evidence of successful checkout. The policy must distinguish an unavailable measurement from an acceptable result. If the service cannot tolerate continued exposure without that evidence, holding the current traffic share is not enough. The response may need to withdraw the candidate while the evidence path is investigated.
A hold is particularly easy to misuse when both populations deteriorate together. The gate should block expansion, but a similar failure rate on v1 and v2 does not identify a safe destination. A shared dependency could be failing independently, or v2 could have increased pressure on it. This needs a service incident response, including a decision about ongoing exposure. Withdrawing v2 might help; it should not be assumed to fix the shared problem.
Apparently conflicting signals need the same discipline. Healthy instances and failing customer outcomes are not competing votes to average into a decision. They describe different things. For a checkout, evidence that customers cannot complete the promised journey takes priority over reassuring infrastructure readings. If only one customer population is suffering, a healthy aggregate does not make that harm acceptable. Investigate the population and attribution while acting on what is already known about its impact.
Automating the response we actually intend
A release controller can collect measurements at defined points, evaluate acceptance and failure conditions, then permit a stage, block progression or initiate an abort. It can retain the observations and decision reason for investigation, and require a named reviewer when the result is inconclusive. The engineering policy specifies what those actions mean. Automation makes their execution repeatable; it does not decide which consequences the business should accept.
Argo Rollouts provides one concrete implementation. Its analysis runs query external providers and can produce successful, failed or inconclusive results. The documented canary background analysis example aborts on failure and sets the canary weight back to zero. An inconclusive analysis can pause progression for intervention. Missing values and measurement errors need explicit handling in the actual configuration. These mechanisms are useful examples, not a contract shared by every release controller.
Argo's analysis documentation describes these outcomes and provider result handling. The paper scene uses its own illustrative policy, not Argo defaults.
The boundary conditions deserve as much testing as the ordinary success path. Feed the gate a known regression, an empty response, a stale observation and an unavailable provider. Check whether it blocks promotion, records why and carries out the required response. Then check what happens if the route update itself fails. Preserving an analysis result is useful, but the system also needs to surface a failed containment action rather than quietly leave the defective share in place.
Conditions that are too sensitive can abort sound releases because of ordinary variation or unrelated faults. Conditions that are too forgiving can permit substantial customer harm. Choose boundaries from the service's acceptable behaviour, the severity of failure and the reliability of the evidence, then test the assumptions behind them. A comparison with v1 can help distinguish a regression, but absolute customer limits still matter. There is no universal failure percentage that makes a payment journey safe.
The costs of a false decision are not symmetrical. An unnecessary abort can delay a useful change and create operational work. Missing a serious defect can expose many more customers before anyone intervenes. A particularly harmful event may justify stopping without waiting for a rate to cross a boundary. The policy should account for that consequence rather than treating every failure as one interchangeable counter.
Continued monitoring matters after a permitted promotion too. A stage can meet its conditions and still miss a failure that appears at greater load or during later work. Permission to move to the next share is not approval to complete the release regardless of what happens afterwards. The different repaired candidate in the illustration has met its stipulated conditions for one stage; it has not acquired a guarantee of safety.
Google's canarying guidance connects evaluation to progression and discusses the trade between detecting bad changes and rejecting good ones.
A person is useful when judgement is needed
Human review can help when evidence conflicts, a dependency incident complicates attribution or the proposed response has risks the controller cannot evaluate. The reviewer needs the relevant observations, the current exposure and a clear question to resolve. An approval button adds little protection if the engineer only checks that the deployment job is green.
Give that review to someone able to act, with a clear handover if they become unavailable. A paused release should not wait unnoticed at the end of a shift. If an override is necessary, make it deliberate and visible: record who authorised it, the reason, the exposure permitted and the checks that will follow. An exception must not quietly turn off the remaining safeguards. Where a response is blocked, the owner needs to know that customer exposure is continuing, not just that an analysis failed.
For Fold & Field, the established confirmation evidence is already sufficient to reject further exposure. We stop progression and, after checking the stable destination and confirming the routing change, stop directing new checkout attempts to v2. That prevents more newly routed attempts from encountering this defect. It does not settle what happened to the attempts that already passed through it.
Orders may have been saved. Payments may have been recorded. Fulfilment messages may already be waiting for processing. Stopping a release prevents further exposure. What happens to the work it has already done?
Recovering the application is only part of the work
Rollback isn't rewind
New checkout requests are reaching v1 again. Once the routing change takes effect, those requests no longer encounter the defective confirmation code. Requests already running through v2 may still finish or fail, but withdrawing it from new traffic limits further exposure. It doesn't change what happened to the purchases it already handled.
For one affected purchase, the order was saved, payment was recorded and a fulfilment message was accepted. Then the confirmation page failed. The fallback returned HTTP 200 without a useful confirmation, leaving the customer unable to tell whether the purchase had succeeded. Returning to the old application doesn't remove any of those records or finish the customer's journey.
Following that purchase means crossing several boundaries. The order database has its own record, the payment provider has another and the broker has accepted work for a downstream consumer. A durable purchase identifier can link them, alongside the provider's operation reference and the message identifier. A request correlation identifier helps trace one attempt, but retries may create new requests for the same intended purchase. We need to recognise the purchase across those attempts too.
Fold & Field / One affected purchase
What didn't roll back?
Purchasepurchase-42After withdrawal: new traffic reaches v1
Requests already running are not cancelled by this switch.
Lift the application layerThe purchase underneath remains
The same purchase, across four boundaries
Order database / order-314
The order was saved.
Saved
The stored order links to purchase-42. Changing the application route does not delete or repair it.
Still to establishWhether later business processing completed still needs checking.
Payment provider / payment-72
Payment was recorded.
Recorded
The payment record belongs to this purchase. A traffic change does not cancel the external operation.
Still to establishSettlement, refund and any later provider state have not been established by this example.
Message broker / message-81
The fulfilment message was accepted.
Accepted
Acceptance establishes that the message entered the messaging system. It does not establish successful fulfilment.
Still to establishThe message may be pending or already handled. The consumer and business outcome must be reconciled.
Customer's browser / purchase-42
The confirmation is missing.
HTTP 200, no useful confirmation
After the previous operations succeeded, the confirmation builder failed. A fallback returned HTTP 200 without a useful confirmation.
Still to establishThe customer cannot reliably tell from this page whether the purchase succeeded.
Each record refers to the intended purchase purchase-42. Matching identifiers connect the evidence; they don't repair it.
Those records establish different things. Broker acceptance doesn't prove that a worker has completed fulfilment. In RabbitMQ, for example, a publisher confirmation and a consumer acknowledgement cover different parts of delivery. Even an acknowledgement is only as meaningful as the point at which the application sends it. The business outcome still needs checking. Similarly, a recorded payment doesn't establish every later financial state, including settlement or refund. The browser, order service and downstream systems can hold different, individually accurate accounts of one unfinished journey.
RabbitMQ's acknowledgement documentation distinguishes publisher confirms from consumer acknowledgements. Neither is a universal receipt for completed business work.
When an uncertain customer tries again
A customer who sees no confirmation may submit the purchase again. If every submission creates a new order and payment operation, a defect in presentation can become duplicate business activity. Recognising a retry requires an identifier that stays attached to the intended purchase, rather than generating a fresh identity each time the browser sends a request.
This is where idempotency becomes useful: repeating the same operation should not repeat its business effect. The checkout can persist the purchase identifier, processing state and result, then use an atomic claim or uniqueness constraint to stop concurrent retries creating separate orders. A later attempt can return the existing result or report that processing is still unresolved. The identifier must refer to the same purchase details; a different intended purchase needs its own identity. Persisting a key without coordinating access is not enough; two requests could both check for its absence before either writes it.
A payment provider may support an idempotency key for its own operation. Reusing the key within that endpoint's documented contract can prevent a retry from creating another payment. That protection has a scope and retention rules. It doesn't automatically cover the order database or the fulfilment consumer, and it doesn't make the whole distributed checkout happen exactly once.
If a payment request times out, the provider may already have acted. Creating another operation with a new key could charge the customer again. Recovery needs to establish the existing operation's status, using its durable reference or the provider's supported retry and lookup mechanisms, before deciding what to do next. The same principle applies to fulfilment: an uncertain response is not evidence that nothing happened.
Stripe's explanation of idempotency explores ambiguous failures. Its API contract defines the behaviour and limits of its supported requests.
There is another boundary between saving an order and publishing its message. In a different failure, the database write could commit while publication fails. A transactional outbox addresses that gap by storing the order change and a message intent in the same database transaction. A relay publishes the stored intent afterwards. Publication can still be repeated, so consumers need a way to recognise duplicate business work. The pattern doesn't put an external payment inside the database transaction or prove that fulfilment succeeded. In our established incident, the message was accepted; we still need to discover what happened after that.
AWS's transactional outbox guidance explains the database and publication boundary, including duplicate delivery.
The old version must understand what remains
The confirmation defect didn't involve a database migration. Consider a separate change in which v2 replaces a record field or changes a message format that v1 still needs. Returning traffic to v1 would restore the old code, but it could fail when reading the new data. Blue/green environments and canary routing don't provide compatibility between those versions.
An expand, migrate, contract sequence keeps that transition manageable across releases. First, expand support so readers understand both representations while writers keep using the old one. Then migrate writers and existing records, preserving what old readers need for as long as they remain in use or might be restored. In a field rename, that might mean maintaining both fields consistently during the transition. Every writer, background consumer and relevant queued message matters, not just the current web application.
Only contract the old representation after those readers and the requirement to return to them have been retired. That last step changes the recovery boundary: the original v1 may no longer be a valid rollback target. Additions also need testing against strict validators and older consumers. A format that looks compatible on paper can still change the meaning of a value or leave two writers maintaining different versions of it.
Martin Fowler's Parallel Change describes the staged approach. AWS's rollback safety guidance examines reader and writer compatibility during deployment.
Separate hypothetical change / Not the confirmation defect
The field changed. The old reader didn't.
{"purchase_ref":"F42"}Original v1 requires order_ref. It cannot read this record.
Unfold a compatible sequence
Expand
{"order_ref":"F42"}Prepare readers to accept either field. Writers still use the old field, and old v1 remains compatible.
Original v1: can read
Prepared reader: can readMigrate
{"order_ref":"F42","purchase_ref":"F42"}Populate the new field and keep both values consistent while old readers or rollback targets still need the old one.
Original v1: can read
Prepared reader: can readContract
{"purchase_ref":"F42"}Remove the old field only after its readers and rollback requirement are retired. Original v1 is no longer a valid return target.
Original v1: cannot read, now retired
Prepared reader: can read
Recovering the purchase
For Fold & Field, recovery starts with identifying the attempts that reached the faulty confirmation path. Reconcile their orders with the payment provider's records, then establish whether fulfilment completed, remains pending or needs investigation. Keep unresolved cases visible rather than treating an absent event as proof of failure. Retrying everything or replaying every message could repeat work that already succeeded.
Where a purchase is verified, the team can restore its confirmation or provide a reliable order lookup. Other cases may need an engineer or support colleague to investigate before work resumes. A refund, cancellation or replacement operation depends on what the business has actually committed and what the customer needs. None follows automatically from the fact that the confirmation page failed.
The application is recovered when it can serve the intended version reliably. The affected purchase is recovered when its state is understood and the customer has a trustworthy outcome. Designing for compatible versions, identifiable operations and safe retries makes that second task possible. The final chapter turns to how teams make these release practices dependable across an organisation.
Shared capabilities, different responsibilities
Making safe releases routine
Fold & Field has stopped sending new checkout requests to the defective version and identified the recovery work its purchases need. That doesn't establish that every customer case is resolved. It does reveal how much a dependable release requires beyond a working deployment pipeline.
Now imagine the shop has grown into dozens of services maintained by different teams. Payments, product images, fulfilment and internal systems all need changes, but those changes carry different consequences. Having each team build its own routing controls, analysis integration and withdrawal process would duplicate work and leave important differences to chance. Requiring the full checkout release process for every service would create a different problem: cost and friction without a clear benefit.
A shared way to release
A platform can provide a consistent way to identify the running version, inspect deployment state and use supported release controls. Where a service needs version aware routing, the team should be able to request it without assembling the controller and routing integration from scratch. Evidence collection, release analysis and decision records can connect to the same experience, so an engineer can see why progression was permitted, held or aborted.
The useful abstraction is an understandable operation. A developer should be able to withdraw new exposure without knowing every implementation detail of a service mesh, while still seeing the intended route, its effective state and any failure to change it. The platform needs to report a failed containment action clearly, retain the evidence and offer a documented way to intervene. A simple interface that hides continuing customer exposure has removed information the engineer needs.
That experience also needs support. Teams should know who maintains the release machinery, who responds when it fails and what the application owner must do during an incident. Access to supported strategies can be self service without making operational responsibility anonymous. The controller is part of the production system too: unavailable measurements, lost permissions and failed integrations need tested responses, just as application failures do.
The CNCF member article on internal developer platforms describes reducing complexity while preserving context. The release capabilities here are a proposed application of that principle, not a prescribed platform architecture.
Different workloads, different controls
Sharing the machinery doesn't share the business knowledge. The checkout team still has to define a successful purchase, supply a trustworthy confirmation indicator and establish the conditions for continuing exposure. The platform can evaluate that indicator; it can't derive it from CPU usage or successful HTTP responses. Compatible versions, safe retries and a plan for uncertain transactions remain application concerns, even when the platform supplies the mechanisms that support them.
For this checkout, controlled production exposure, outcome checks, retained stable capacity and verified withdrawal address worthwhile risks. They work together only if the application supports them. A route back to v1 is useful when v1 can handle the load and understand the state it will encounter. Those requirements belong in the supported release path, with the application team verifying its part rather than assuming the platform has made it true.
Fold & Field / Supported paths for future changes
One platform, different releases.
Controlled production exposure
Customer outcomes inform progression and withdrawal.
Platform providesA supported route, retained capacity and an analysis integration can connect evidence to a verified withdrawal of new requests.
Application suppliesThe checkout team supplies the confirmation indicator, acceptance policy, compatible versions and safe handling of purchases and retries.
The boundaryWithdrawing new traffic does not resolve the purchases already handled.
Rolling replacement
Availability controls govern replacement. Customer checks still matter.
Platform providesA supported rolling update replaces copies with the required availability controls and reports what is running or blocked.
Application suppliesHere, the service serves replaceable image variants from preserved originals. Failures are easy to detect, reversal is tested and customer impact is limited.
The boundaryCopy replacement is not an evidence based promotion gate. A service that deletes originals or exposes private images needs a different risk assessment.
Controlled consumer activation
Explicit activation controls new work. Running operations may continue.
Platform providesA supported worker path controls candidate consumer activation and reports its effective state. It does not use an HTTP weight to hold work back.
Application suppliesThe worker team supplies compatible message handling, processing outcomes, duplicate prevention and a recovery plan for work already accepted.
The boundaryStopping new consumption does not necessarily stop work already running. Broker and consumer behaviour must be verified.
The image service illustrates why checkout's release path doesn't need to be the only option. In this example, it serves replaceable image variants from preserved originals, failures are visible and reversal has been tested. Customer consequences are limited, so a conventional rolling replacement with appropriate availability controls may be enough. That reasoning would change for a service that deletes originals or could expose private images. The workload's name doesn't determine its risk.
The fulfilment worker needs a different control altogether. Holding its HTTP traffic at zero doesn't prevent it from consuming messages. Its supported path needs to govern consumer activation, expose the actual processing state and distinguish stopping new work from stopping work already running. The application team must understand message compatibility, duplicate handling and recovery. Common platform capabilities can support all three services without pretending that exposure means the same thing for each.
Making the capability earn its cost
Each supported path becomes something the organisation must maintain, upgrade and understand when it fails. Retaining two complete application environments consumes capacity. Analysis depends on reliable signals. Routing controllers and their integrations add more ways for an intended action to fail. Those costs can be justified, but a larger collection of deployment features is not evidence of better delivery.
DORA's software delivery measures provide a useful view of throughput and instability: change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate. They examine how changes reach production, how often deployments need immediate intervention and how much delivery follows from incidents. Read together, they can reveal improvement that a deployment count alone would miss.
Examine those measures over time for the service being changed, rather than ranking unlike teams. A checkout and a quiet internal system have different constraints. Alongside delivery performance, Fold & Field should examine how quickly harmful changes are detected and contained, the scale and duration of customer impact and whether affected journeys are recovered. Deployment recovery time alone doesn't establish that every purchase has been reconciled.
DORA's current guidance describes five delivery measures and warns against single metric targets and comparisons between dissimilar applications.
The platform also needs evidence from its users. Can teams release ordinary changes without specialist help? Where do they wait or need intervention? Consider that experience alongside adoption, infrastructure spend and the operational effort needed to keep the delivery machinery working. A platform that slows routine changes without reducing meaningful harm has not justified its complexity.
Fold & Field doesn't need to answer one incident by building an enormous platform. It can establish the analysis integration, decision record and reliable withdrawal mechanism with checkout, verify their behaviour and offer them to teams facing similar problems. Test missing evidence and failed route updates before treating that path as dependable. Support the teams using it, learn where the interface hides something important and improve the capability through actual use. Simpler services can keep a simpler supported path.
The original deployment succeeded while customers were left uncertain about their purchases. Across the release, a healthy process, a controlled traffic share and a route back to old code each answered only part of the problem. None could establish the customer outcome on its own.
Dependable delivery brings those parts together: an application whose behaviour is understood, evidence tied to what customers need, controls that work and a practical way to recover operations already affected. The appropriate process is the simplest one that can manage those consequences reliably. Its success is visible in changes reaching customers with risks understood, problems recognised and recovery made possible, rather than in the arrival of a new version alone.
Read the engineering underneath
References and model notes
Technical references checked on 6 November 2026. The shop, evidence packets, policy thresholds and operational numbers are fictional. Calculations are deterministic expectations under the assumptions beside each experiment. They are not measurements or a validated production simulator.
- Google SRE Workbook: Canarying ReleasesComparing populations, isolation and the limits of artificial traffic.
- Argo Rollouts: Analysis and Progressive DeliveryAnalysis outcomes, pauses and decisions when evidence is inconclusive.
- Argo Rollouts: Canary Deployment StrategyReplica approximation and explicit traffic routing.
- Argo Rollouts: BlueGreen Deployment StrategyActive and preview environments, switching and propagation delays.
- AWS Builders’ Library: Ensuring rollback safety during deploymentsCompatibility between readers and writers across version changes.
- Google SRE Workbook: Implementing SLOsUser focused reliability targets and error budgets.
- Google SRE Book: Monitoring Distributed SystemsInternal health, externally visible behaviour and latency signals.
- Microsoft Well-Architected: Safe deployment practicesRisk, release health checks and operational preparation.
- Stripe API: Idempotent requestsHow an explicit request key can prevent a repeated operation. Provider rules and retention limits still apply.
The seven chapters and technical explanations are available without JavaScript. Controls enhance the examples. With JavaScript disabled, comparisons and explanatory outcomes remain readable.