Payment Failover: How to Survive a Provider Outage

If a provider is failing on you right now, the practical triage is in what to do when a payment gateway goes down. Read that first if the site is on fire. Come back here when you want to make sure the next outage is boring.

A failed payment is not one event. It is three, and only one of them is a failover case.

• Technical failure — a timeout, a connection reset, an HTTP 500, a provider outage. Nobody made a decision about this payment. This is the failover case.
• Hard decline — insufficient funds, a stolen card, a closed account. The issuer decided, and the issuer is the same institution no matter which provider asks.
• Soft decline — a risk score, a velocity rule, an unusual cross-border pattern. Another provider sometimes presents it differently and it clears. A judgement call, not a rule.

Getting this wrong in the safe-looking direction is expensive. Retry a hard decline through four providers and you have not recovered a sale — you have shown the customer's bank a pattern that looks exactly like card testing, and given your own provider a reason to review your account. The full breakdown of the three categories, with what to tell the customer in each case, is in why payments fail.

Choosing where a payment goes next is a lookup. Somewhere you hold a list of the providers that can take this currency and this payment method, in an order you control, and you walk down it. That logic is a morning's work.

What makes it hard is everything the second provider does not share with the first. A different authentication scheme. A different set of error codes, with a different opinion about which of them are retryable. A different webhook signature format. A different definition of when a payment is final. Adding a fallback provider is not adding a fallback — it is adding a second integration, and then a third thing that keeps both of them consistent. That third thing is the actual product, and deciding whether to build it is the subject of what payment orchestration is.

Here is the sequence that turns a failover feature into a liability. It is not an edge case. It is the normal failure mode of a network.

1. Your checkout sends a charge to provider A.
2. Provider A receives it, authorises it, and charges the customer's card.
3. The response never arrives. A load balancer drops it, a timeout fires, a container restarts.
4. Your system sees no answer. As far as it knows, the payment failed.
5. Failover does its job and sends the payment to provider B.
6. Provider B charges the card. The customer has now paid twice.

Nothing in that sequence is a bug. Every component behaved correctly. The failure is architectural: the payment had no identity that outlived a single HTTP request, so there was no way to ask did this already happen? before trying again.

The fix is an idempotency key, and the detail that decides whether it works is what the key is attached to. A key generated per HTTP request protects nothing — the retry is a new request, so it gets a new key, and the second charge goes through. The key has to identify the payment attempt: derived from your own order reference, stored with the order, and reused by every retry of that attempt, including the one that goes somewhere else.

Which produces the requirement most people miss on a first build: the key cannot belong to a provider. Provider A's idempotency key means nothing to provider B. If your only protection is the provider's own key, then the exact moment you need it — crossing from one provider to another — is the exact moment it stops working. The identity has to be held one level above both of them, an argument developed further in how payment orchestration works.

There is a second half to the same problem. After a timeout you do not know the outcome, and guessing costs money in both directions: assume success and you ship goods nobody paid for, assume failure and you either lose the sale or charge twice. Something has to go back to the provider and ask what actually happened to that payment rather than inferring it from a dead connection. Doing that across several providers at once is its own exercise — why the numbers never match is what it looks like when nobody does.

There are three places to put failover logic and two of them are wrong.

Inside a provider integration is wrong because it cannot see the other providers. In your checkout is wrong for a subtler reason: it makes every change to your failover behaviour a deployment. The moment you most want to move traffic away from a provider is the moment they are having an incident, and shipping code to production during someone else's incident is how a bad hour becomes a bad day.

The third place is a layer between your checkout and every provider you run. It is the only position that can see all of them at once, hold one identity for a payment across all of them, and be reconfigured without touching your application. That is the structural argument for orchestration, and the same argument that makes switching providers a configuration change rather than a project.

Rerouting assumes you know a provider is unhealthy. That judgement is harder than it sounds, because the two obvious approaches are both wrong in opposite directions.

Reacting to a single failure makes the system flap. One timeout is not an outage — networks lose packets, providers have slow seconds, and a layer that reroutes on the first error will move traffic away from a perfectly healthy provider constantly, which loses you the approval-rate advantage that provider had and makes behaviour impossible to reason about.

Waiting for certainty fails the other way. By the time a provider has failed enough times to be unambiguously down, every one of those failures was a real customer who did not pay. The whole value of failover is spending a small number of failed attempts to learn something, then not spending any more.

What works in between is a rolling window with a threshold and a cooldown. Count technical failures only — declines are the provider working correctly and must never count toward its health — over a short window. Cross the threshold and the provider is marked unhealthy, so subsequent payments route elsewhere without paying the timeout cost again. Then, crucially, let it back: after a cooldown, send a small share of traffic to it and restore it fully if that succeeds. A provider that is marked down and never re-tested is a provider you have permanently lost, usually after a fifteen-minute incident.

Two details matter more than the numbers you pick. Health must be tracked per provider, not globally, or one bad provider degrades routing for all of them. And the timeout you set on the outbound call is itself part of detection: too long and each failure costs the customer thirty seconds of spinner before you even begin the second attempt, which is often worse for conversion than the outage.

Failover is narrow, and overselling it produces worse systems than not having it. It addresses exactly one failure mode: the provider could not give you an answer. Three things it is regularly and wrongly expected to solve are worth naming.

It does not fix declines. The issuer, not the acquirer, decides whether a card is approved, and the issuer is the same institution no matter which provider carries the request. Retrying a stolen or expired card through a second provider will fail again, cost a second fee, and add another failed authorisation to that card's history. The categories and what each deserves are in why payments fail.

It does not carry stored payment methods across. A token belongs to the provider that issued it. If a customer's saved card lives at provider A and A is down, provider B cannot charge it — it has never seen that card. For one-off checkouts this is invisible because the customer is present and can enter details again. For renewals it is decisive, and it is why a business built on recurring revenue needs its second provider collecting agreements from the start rather than standing by unused.

It does not survive an account problem. If a provider suspends your account rather than having an outage, failover works exactly as designed and keeps working — but only if the second provider is also live and underwritten. This is the case that catches businesses in elevated-risk categories, where the interruption is commercial rather than technical.

Failover is one of several jobs a payment layer performs, and it is easier to size correctly next to the others — routing, idempotency, verification and reconciliation — in what payment infrastructure has to do.

Whether you build this or buy it, these are the questions worth answering out loud. Most implementations answer the first two and stop.

• Which specific error codes from each provider count as retryable, and who revisits that list when a provider changes their API?
• What is the identity of a payment attempt, and does it survive being handed to a second provider?
• After a timeout, what goes back and asks the provider what happened — and how long does the order sit in an unknown state first?
• Can you move traffic away from a provider without a deployment?
• Does your reporting distinguish one payment attempted twice from two separate payments?
• Have you tested it? Not the happy path — the timeout, against a provider that succeeds and does not tell you.

The last one is the one nobody does before launch, which is why it belongs on a pre-launch checklist rather than in a post-mortem.

PaymentHood is the layer described two sections ago. Your checkout integrates once — through a free plugin or the API — and every provider you enable sits behind it, so the providers you can fall back to become an account setting rather than a build. Moving the default provider for a currency is a change in a dashboard, which means you can do it while a provider is having an incident without shipping anything. On card payments the provider is selected for you rather than put to the shopper, and that selection has an order behind it.

Layer three is handled where it has to be. Charge creation is idempotent on your own order reference, checked before any provider is contacted — so the identity of a payment lives above the providers instead of inside one of them, exactly as the double-charge problem demands. Webhook signatures are verified centrally and orders are confirmed server-side rather than from a browser redirect, so a second provider is not a second chance to get either of those wrong.