Building a failover you can actually trust, one health check at a time

A load balancer goes in front of a cluster's API servers. It is the newest thing in the path, installed by whoever was brought in to install it, and the first report back is that traffic sent to it never reaches the backend. The status page shows the pool down. Nobody disputes the symptom.

What follows is usually weeks rather than hours. The people who installed the balancer can show it is configured correctly. The people who own the cluster can show their service is running. The network team can show the rules it was asked for were applied. Each reads a different instrument, each instrument says the problem is elsewhere, and the one they have in common — the balancer's own health check — is being read as a red light rather than as evidence.

It is evidence. A health check knows considerably more about why it failed than most people think to ask it, and asking is the difference between a bounded conversation and a standoff.

A health check is a diagnostic, not a light

When a checker tries to open a connection to a backend and cannot, there are two distinct ways that attempt fails, and they do not mean the same thing.

The first is an immediate refusal. The attempt reaches the host, nothing is listening on that port, and the host says so at once — the rejection comes back in effectively no time. That is a precise statement: the machine is up, the path to it works, and the service you want is not there.

The second is a timeout. The attempt goes out, nothing comes back, and the checker gives up when its connect timeout expires — the elapsed time is the timeout, near enough exactly, every time. Something between the checker and the service is discarding traffic without answering: a firewall rule, a missing route, a policy that drops rather than rejects.

Both appear in the summary as the same red backend. One means the platform team has a service that is not running; the other means the network team has a rule that is not what anyone believes it is. The elapsed time, which most checkers will show you, separates them for free.

Build the habit, because where several parties sit in the path the health check is the only artifact all of them can see at once. Read properly it says, with evidence, whose next hour this is; read as a light it says nothing anyone can act on.

A test that passes without the service is not a test

The second thing that goes wrong is verification that cannot fail for the right reason.

It usually starts helpfully. Someone wants to prove the path works before the real service is ready, so they stop the service and put a small hand-rolled listener on its port. The path is now proved — against the listener. Then the listener gets left suspended in a shell rather than killed, still holding the port but accepting nothing, and the checker correctly marks the pool down. What it is reporting on is no longer the service anybody cares about.

The quieter version is testing the wrong door. A reachability check gets run against the balancer's management interface rather than the port that forwards traffic. That interface speaks plain HTTP, the test speaks TLS, and the protocol-mismatch error that comes back says nothing about the forwarding path — then circulates as evidence about it, because it looks severe and came from the right machine.

Both are the same defect: the outcome of the step does not depend on the thing the step claims to verify.

So write the expected output beside every verification step before running it — this command, against this address, must print this. A step you cannot write an expected output for is not a test, and one that would pass with the real service absent is worse than no step, because it gets quoted later.

The redundant thing is a single point of failure too

A balancer in front of several API servers makes the servers redundant. It does not make the path redundant: one instance, one machine, one configuration file — the shape the component was bought to remove.

The standard answer is a pair with a floating address that lives on whichever instance is healthy. Three details decide whether it works when it matters.

The promotion check has to test the service, not the host. A machine that is powered on while the forwarding process is dead is the precise failure the pair exists to survive, and a check that only proves the host is alive keeps the address exactly where it does no good. Ask both questions: the process is running, and it is accepting connections on the port it is meant to serve.

It needs hysteresis. A check that demotes on a single miss and promotes on a single pass will hand the address back and forth across an unremarkable blip. Require a few consecutive failures before giving up a role, and the same before taking one back.

And the management interface is part of this. The page showing which backends are up is the control surface for your own availability: bind it to loopback or a management network, reach it over a tunnel, give it a real credential, and never let it answer on the public address.

How this gets handled in practice

In a managed cloud environment most of the above is a property of the service rather than a decision. Redundancy across failure domains comes with the balancer, the health endpoint is a documented contract, and the operator's instrument is a dashboard they did not configure and cannot misconfigure.

Across Pakistan and the wider region the same role is more often a self-managed instance on a virtual machine inside an institution's own network — cheap, well understood, already where the regulated workload sits, and requiring no cloud footprint anybody has to approve. What comes with it is an ownership boundary the managed version does not have: platform, network and application commonly sit with three different parties, often including more than one external vendor, and no single party can see end to end. Diagnosis travels by email between organisations that each have reason to believe the problem is somewhere else.

Which inverts the usual assumption about who needs good instruments. An unreadable health check in a managed environment is an inconvenience; in a multi-party on-premise estate it is what turns a configuration question into a month of correspondence.

Before the next change

Write the expected output beside every verification step, and delete any step that would pass with the service switched off. Learn the two failure modes your checker reports and put both in the runbook with what each means and whose problem it is. Make the promotion check prove the service is accepting connections, not that the machine is powered on.

And settle in writing whether what you have is a pair or a single instance. A single instance is a defensible decision with a cost attached. What is not defensible is a diagram showing two.