Why Integrations Break Quietly

· 5 min read

Connecting two systems is easy to demonstrate and hard to keep working. The demonstration takes an afternoon: authorise the connection, watch a record appear on the other side, and everyone agrees the problem is solved. What follows is a slow divergence that nobody notices until a customer points it out.

The reason is that integrations rarely fail in a way that produces an alarm. They fail in ways that look like success.

Failure one: the credential that expired months ago

Most connections between business systems are authorised rather than passworded, and that is the right design. The framework almost everyone uses exists so that an application can "obtain limited access to an HTTP service, either on behalf of a resource owner by orchestrating an approval interaction between the resource owner and the HTTP service, or by allowing the third-party application to obtain access on its own behalf" [2]. Limited access, granted deliberately, revocable without changing a password — all genuinely better than handing over credentials.

The catch is that these grants are designed to end. Tokens expire. Permissions get revoked when someone leaves. A provider changes its policy and shortens a token's life from permanent to sixty days. None of these events look like an outage. The integration simply stops being allowed, and what it does next is entirely up to how it was written.

The well-written ones surface it. Many do not. They log a failure into a file nobody reads, return a success to the interface, and continue on a schedule for months. The dashboard stays green. The data stops moving. The two systems drift apart at exactly the rate your business generates new records.

The practical defence is to distrust any connection status that was determined by asking the connection about itself. A status that says "connected" usually means "we were connected when this was last saved". The only trustworthy check is a real operation performed now.

Failure two: the retry that duplicated everything

The opposite failure is an integration that works too eagerly.

Networks are unreliable in a specific, awkward way: a request can succeed on the server and still fail to deliver its answer. From the sender's side, "it worked but I did not hear back" is indistinguishable from "it never arrived". The natural response is to retry, and the natural result is two invoices, two contacts, or two charges.

The formal property that prevents this is idempotency: "A request method is considered 'idempotent' if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request" [1]. In practice this means each operation carries a key the receiver remembers, so a repeat is recognised as the same request rather than a new one.

This matters for anyone evaluating a tool, not only for the people building it. It is a fair question to ask a vendor: if your system sends the same thing twice because a network dropped a response, what happens on our side? The answers divide neatly into those who have thought about it and those who will discover it in your account.

Failure three: the field that quietly stopped mapping

The third failure needs no technical fault at all.

Someone renames a field. Someone adds a required field to a form. Someone changes a dropdown from "Customer" to "Client". The integration keeps running, keeps reporting success, and keeps writing into a field that no longer means what it meant. Six weeks later a report is wrong, and the cause is a change that was correct, deliberate, and entirely reasonable in the system where it was made.

This is the failure that no amount of engineering discipline on either side prevents, because neither side is broken. The mapping between them is what broke, and the mapping usually lives nowhere — not in a document, not in a diagram, sometimes not even in a person's memory.

The pattern underneath

All three share a shape: the integration reports on its own health, and its self-report is not evidence.

The corrective is to verify the outcome rather than the mechanism. Not "did the sync run" but "does the record on the other side actually match". Not "is the connection authorised" but "did an operation succeed in the last day". Anything that can be answered by a status flag can be answered wrongly by a stale status flag.

There is also a design argument for connecting fewer things. Every integration is a promise to keep two representations of the same fact in agreement forever, and that promise gets harder as both systems evolve independently. A tool that holds the customer record and the messages and the invoices in one place has no mapping to maintain between those three, because there is nothing to map. That is a less exciting claim than most integration marketing, and a more durable one.

There is a fourth failure worth naming briefly, because it is the one that produces the strangest symptoms: partial success. An integration that writes the customer but fails on the address, or creates the invoice but not its line items, leaves behind a record that exists and is wrong. That is worse than nothing, because nothing prompts someone to investigate while a half-written record quietly gets used.

What to do about the ones you already have

Write down what each integration is for, in a sentence, in a place a person will find. Most organisations cannot produce this list, which is itself the finding.

For each one, decide what evidence would show it working — a record created, a field matching, a timestamp advancing — and check that evidence on a schedule rather than checking the status page.

Finally, be sparing about what flows. Every field an integration copies is a field that can silently diverge, and a field that nobody reads is pure liability. The habit of not collecting data speculatively "in case it is useful later" [3] applies just as well to data you copy between systems: if no one would notice it missing, moving it is a maintenance cost with no return.

An integration that moves three fields correctly is worth more than one that moves forty, thirty-six of which no one has looked at since the day it was built.

Sources

  1. [1] RFC 9110 — HTTP Semantics, §9.2.2 Idempotent Methods — IETF
  2. [2] RFC 6749 — The OAuth 2.0 Authorization Framework — IETF
  3. [3] 360REV Use of Data Policy — what we collect — 360REV, Inc.