Payment Failover: How to Survive a Provider Outage

If a provider is failing on you right now, the practical triage is in what to do when a payment gateway goes down. Read that first if the site is on fire. Come back here when you want to make sure the next outage is boring.

A failed payment is not one event. It is three, and only one of them is a failover case.

• Technical failure — a timeout, a connection reset, an HTTP 500, a provider outage. Nobody made a decision about this payment. This is the failover case.
• Hard decline — insufficient funds, a stolen card, a closed account. The issuer decided, and the issuer is the same institution no matter which provider asks.
• Soft decline — a risk score, a velocity rule, an unusual cross-border pattern. Another provider sometimes presents it differently and it clears. A judgement call, not a rule.

Getting this wrong in the safe-looking direction is expensive. Retry a hard decline through four providers and you have not recovered a sale — you have shown the customer's bank a pattern that looks exactly like card testing, and given your own provider a reason to review your account. The full breakdown of the three categories, with what to tell the customer in each case, is in why payments fail.

Choosing where a payment goes next is a lookup. Somewhere you hold a list of the providers that can take this currency and this payment method, in an order you control, and you walk down it. That logic is a morning's work.

What makes it hard is everything the second provider does not share with the first. A different authentication scheme. A different set of error codes, with a different opinion about which of them are retryable. A different webhook signature format. A different definition of when a payment is final. Adding a fallback provider is not adding a fallback — it is adding a second integration, and then a third thing that keeps both of them consistent. That third thing is the actual product, and deciding whether to build it is the subject of what payment orchestration is.

Here is the sequence that turns a failover feature into a liability. It is not an edge case. It is the normal failure mode of a network.

1. Your checkout sends a charge to provider A.
2. Provider A receives it, authorises it, and charges the customer's card.
3. The response never arrives. A load balancer drops it, a timeout fires, a container restarts.
4. Your system sees no answer. As far as it knows, the payment failed.
5. Failover does its job and sends the payment to provider B.
6. Provider B charges the card. The customer has now paid twice.

Nothing in that sequence is a bug. Every component behaved correctly. The failure is architectural: the payment had no identity that outlived a single HTTP request, so there was no way to ask did this already happen? before trying again.

The fix is an idempotency key, and the detail that decides whether it works is what the key is attached to. A key generated per HTTP request protects nothing — the retry is a new request, so it gets a new key, and the second charge goes through. The key has to identify the payment attempt: derived from your own order reference, stored with the order, and reused by every retry of that attempt, including the one that goes somewhere else.

Which produces the requirement most people miss on a first build: the key cannot belong to a provider. Provider A's idempotency key means nothing to provider B. If your only protection is the provider's own key, then the exact moment you need it — crossing from one provider to another — is the exact moment it stops working. The identity has to be held one level above both of them, an argument developed further in how payment orchestration works.

There is a second half to the same problem. After a timeout you do not know the outcome, and guessing costs money in both directions: assume success and you ship goods nobody paid for, assume failure and you either lose the sale or charge twice. Something has to go back to the provider and ask what actually happened to that payment rather than inferring it from a dead connection. Doing that across several providers at once is its own exercise — why the numbers never match is what it looks like when nobody does.

There are three places to put failover logic and two of them are wrong.

Inside a provider integration is wrong because it cannot see the other providers. In your checkout is wrong for a subtler reason: it makes every change to your failover behaviour a deployment. The moment you most want to move traffic away from a provider is the moment they are having an incident, and shipping code to production during someone else's incident is how a bad hour becomes a bad day.

The third place is a layer between your checkout and every provider you run. It is the only position that can see all of them at once, hold one identity for a payment across all of them, and be reconfigured without touching your application. That is the structural argument for orchestration, and the same argument that makes switching providers a configuration change rather than a project.

Whether you build this or buy it, these are the questions worth answering out loud. Most implementations answer the first two and stop.

• Which specific error codes from each provider count as retryable, and who revisits that list when a provider changes their API?
• What is the identity of a payment attempt, and does it survive being handed to a second provider?
• After a timeout, what goes back and asks the provider what happened — and how long does the order sit in an unknown state first?
• Can you move traffic away from a provider without a deployment?
• Does your reporting distinguish one payment attempted twice from two separate payments?
• Have you tested it? Not the happy path — the timeout, against a provider that succeeds and does not tell you.

The last one is the one nobody does before launch, which is why it belongs on a pre-launch checklist rather than in a post-mortem.

PaymentHood is the layer described two sections ago. Your checkout integrates once — through a free plugin or the API — and every provider you enable sits behind it, so the providers you can fall back to become an account setting rather than a build. Moving the default provider for a currency is a change in a dashboard, which means you can do it while a provider is having an incident without shipping anything. On card payments the provider is selected for you rather than put to the shopper, and that selection has an order behind it.

Layer three is handled where it has to be. Charge creation is idempotent on your own order reference, checked before any provider is contacted — so the identity of a payment lives above the providers instead of inside one of them, exactly as the double-charge problem demands. Webhook signatures are verified centrally and orders are confirmed server-side rather than from a browser redirect, so a second provider is not a second chance to get either of those wrong.