Troubleshooting and System Restoration

Recurring faults

A system that keeps failing is not mysterious. It is undiagnosed.

Intermittent faults have causes. They are harder to find than constant faults. They are not different in kind.

A door that releases late twice a month, a panel that drops off the network at 3am and is back by shift change, a camera that records for six days and then quietly stops: each of these leaves evidence behind. Access control event history, controller diagnostics, archiver logs, database job records, switch port counters and Windows event logs all hold traces of what happened and when it happened. Restoration is the work of reading that evidence until the pattern is no longer ambiguous.

The alternative method is replacement. Swap the reader. If the fault returns, swap the controller. If it returns again, swap the power supply, then the lock, then the camera. Each swap is reported as progress, and each one costs an attendance, a work order, an outage window and a piece of hardware. Replacement is a legitimate repair once a component has been shown to be at fault. Used as a way of searching for the fault, it is guessing, and the cost of the guess is carried by the site rather than by the vendor making it.

The test is straightforward. If nobody can say what the fault was, the fault was not found. A closed work order that reads “replaced reader, tested OK” records an action, not a cause. Ask what the evidence was and what it ruled out. Where there is no answer, expect the symptom back.

• Repeat attendances on one symptom. Three or more visits to the same door, panel, server or camera with no written cause on any of them.
• Parts changed with no failure explanation. A component was replaced, but nothing on the record describes how it was shown to be faulty.
• “We rebooted it and it cleared.” A restart that restores service tells you the running state was corrupt. It says nothing about what corrupted it, and it will corrupt again.
• Faults that follow a schedule. Overnight, every Monday, during the backup window, at shift change, after patching. A pattern in time is one of the strongest clues available and is usually the one left unexamined.
• Blame moving between trades. The security vendor points at the network, the network team points at the panel, and nobody owns the measurement that would settle it.
• A workaround that has become procedure. Staff who badge twice out of habit, or prop a door, or check a camera manually every morning. The site has absorbed a fault instead of having it fixed.

How a diagnosis runs

The order matters. Each step either narrows the fault or is discarded, and no component is replaced before the evidence implicates it.

• 01  Reproduce or characterize the fault — Where the fault can be forced, we force it and hold the conditions steady so it can be observed directly. Where it cannot, we characterize it instead: exact times and dates, frequency, which devices, which credentials, which operators, what else was running on site. An intermittent fault that cannot be reproduced on demand can still be bounded, and a bounded fault is a diagnosable fault.
• 02  Gather evidence before changing anything — Access control event history, controller and panel diagnostics, archiver and recording logs, Windows event logs, database and backup job history, switch port counters and error rates, power and UPS records. Evidence is collected first so there is a baseline to compare against. A system that has been reconfigured before it was examined has lost the record of its own failure.
• 03  Build a timeline — Occurrences are aligned against changes: firmware and software updates, Windows patching, network changes, added devices, configuration edits, construction on site, seasonal and temperature patterns. Correlation is not proof, but it discards a large number of hypotheses quickly and it often puts a date on when the system was last known good.
• 04  Isolate the layer — The fault is placed at a layer: field device, cabling and termination, power, network, controller, application, database or server platform. This is done by measurement and by controlled substitution at one layer at a time. It is not done by replacing parts in sequence and waiting to see whether the symptom stops.
• 05  Form one hypothesis — A single statement that accounts for every observed symptom, including the ones that do not fit the convenient answer. A hypothesis that explains four symptoms out of five is not the cause. It is the next thing to disprove.
• 06  Test it in a way that can fail — A real test predicts a result in advance and is capable of coming out wrong. If the hypothesis is a marginal RS-485 run, the test loads or shortens that run and states what should happen. If the prediction fails, the hypothesis is wrong and the work returns to the previous step rather than proceeding on hope.
• 07  Correct the root cause — Repair, reterminate, correct the addressing or the network path, rebuild the service, repair or reindex the database, restore the configuration, or replace the component that has now been shown to be at fault. Replacement still happens. It happens at the end of the process, with a reason attached to it.
• 08  Verify under real conditions — The correction is tested under load and over time rather than once: full card reads at the affected doors, recording verified to disk and against the retention policy, controllers watched through the window in which they previously failed, reconnection and failover exercised where the platform supports it and the site allows a window.
• 09  Document the cause — A written record of the symptom, the evidence gathered, what was ruled out, the cause, the correction and the verification. It is what lets the site hold a vendor to account, what lets the next technician start informed, and what makes a repeat of the same symptom significant rather than mysterious.