Upgrades of enterprise security platforms fail for predictable reasons, and almost none of them are the upgrade itself.
They fail because a third-party integration was written against an older interface. Because the database took four times longer to upgrade than the window allowed. Because the client machines were left at the old version and could not connect on Monday morning. Because a controller firmware level was not supported by the new release, or was supported only after an intermediate step nobody had planned for.
All of those are findable beforehand. What follows is the preparation work that finds them, applicable to Genetec Security Center, to C•CURE 9000, and in substance to any enterprise platform with controllers, a database and a client tier.
Build the compatibility matrix first
There is no single supported version. There is a set of versions that are supported together, and the set is narrower than people expect. Write down the current version of every element and the target version of every element, then confirm each pair against the manufacturer documentation rather than against recollection.
The elements that have to line up: the head-end application version, the database engine version and edition, the server operating system version, the client application version on every workstation, the controller firmware level on every panel, the camera or device firmware where the platform enforces a minimum, and the version of every integration, driver or SDK-based application that connects to the platform.
Two constraints catch people repeatedly. The first is upgrade path. Many platforms will not upgrade directly from a version several releases old, and require an intermediate version as a stepping stone, which doubles the database upgrade time and the window. The second is that clients and servers are usually version-locked to each other. An upgraded server with un-upgraded clients is not a degraded state, it is an outage for every operator who has not been touched.
What to establish before a date is set
| Establish | Why it changes the plan | |
|---|---|---|
| Upgrade path | Whether the current version upgrades directly to the target | An intermediate step can double the window and adds a second point of failure |
| Database engine | Version, edition and whether it also needs upgrading | An engine upgrade is a separate change with its own rollback and its own risk |
| Database size | Size of the event or journal database, and its free space | Upgrade duration scales with size. This is the number the window is built from |
| Express edition limits | Whether the database is on an edition with a hard size ceiling | A journal approaching that ceiling forces an edition change into the same project |
| Operating system | Whether the target release supports the current server OS | An OS upgrade or a new server changes this from an upgrade to a migration |
| Controller firmware | Current level on every panel against the supported range | Controllers may need updating before, after, or in a specific order relative to the head end |
| Client inventory | Every workstation running a client, and who uses it | The forgotten client is always the one at a post that operates overnight |
| Integrations | Every system that connects: visitor management, HR feed, intrusion, elevator, video | Integrations are the single largest cause of post-upgrade breakage |
| Licensing | Whether the entitlement covers the target version and is current | Discovering an expired maintenance entitlement on the night is a stopped project |
| Service accounts | Accounts, passwords and expiry dates for every platform service | A password that expires mid-project produces failures that look like upgrade defects |
| Certificates | Expiry and issuer for server and client certificates | Trust changes during an upgrade surface as clients and controllers refusing to connect |
| Custom content | Custom reports, badge templates, event-to-action rules, scripts | These frequently do not carry forward untouched and need testing individually |
Everything in this table can be established without touching the production system. None of it should be discovered during the window.
Rehearse on a copy. This is the step that pays for itself
Restore a full production database backup to an isolated test instance and run the upgrade there, end to end, before the real one is scheduled. This is not a formality and it is not optional on a large system.
It produces three things nothing else can. First, an accurate duration. Database upgrade time on a multi-year journal is not proportional to anything you can estimate, and the difference between three hours and eleven decides whether the window is a night or a weekend. Second, it surfaces schema and data errors, which on a database that has been through previous upgrades and integrations is a realistic possibility, at a point where they can be worked through calmly. Third, it gives you a working instance of the target version to test integrations, custom reports and badge templates against, weeks ahead of cutover.
The rehearsal also validates that the backup is restorable, which is a separate and useful fact. A backup that has never been restored is a hypothesis. See why configuration backups matter.
Where breakage actually appears
Integrations and interfaces
- Directory or HR synchronization that maps fields which have moved or been renamed.
- Visitor management and badge printing, which depend on photo storage and template paths.
- Video linkage between the access platform and the video platform, where each has its own supported-version list.
- Intrusion, elevator and building-management interfaces running on older protocol versions.
- Anything built on an SDK, which is the category most likely to require a rebuild rather than a reconfiguration.
Server roles and order
- Distributed deployments upgrade in a defined order. Directory or master first, then expansion and archiver roles, then clients.
- Failover and redundant servers need upgrading too, and a failover left at the old version is a failover that will not work.
- Media and archiver roles have their own storage and their own service accounts.
- Virtualized servers need the snapshot taken and, importantly, removed afterwards.
The controller tier
- Firmware updates on controllers such as iSTAR are their own change, with their own risk to the doors they serve.
- Update order relative to the head end is specified by the manufacturer and is not interchangeable.
- A controller mid-firmware-update is a controller not controlling doors. Plan door coverage accordingly.
- Confirm cardholder data is fully downloaded and verified after the panels return, not just that they show online.
The human tier
- Operator interfaces change between major versions. Untrained operators on a changed interface is an availability problem.
- Saved views, layouts and personal settings often do not carry forward.
- Overnight and weekend staff are the ones most likely to meet the new version first and to have no one to ask.
- Brief the control room before the window, not after it.
Verification after the window, in order
The upgrade completing without error is not the acceptance test.
- 01 Services and roles running — Every platform service started, set to automatic, and running under the intended account.
- 02 Controllers online and in sync — Online is not enough. Confirm the cardholder download completed and spot-check a credential at a door.
- 03 Doors tested across types — A sample covering each lock type, each reader type and each controller model. Valid read, invalid read, request-to-exit, door position, held and forced.
- 04 Alarms routing to the right destination — Trigger a real alarm and confirm it presents to the operator with the correct priority and instructions.
- 05 Video linkage — Open the associated video from an access event, both live and recorded.
- 06 Integrations reconnected — Each one individually, with a transaction pushed through it rather than a status light checked.
- 07 Scheduled jobs and reports — Backups, archiving, database maintenance and any scheduled report. These fail silently and are noticed weeks later.
- 08 Badging end to end — Enrol a test record, capture a photo, print a badge, use it at a door, then remove the test record.
- 09 Backup taken at the new version — The pre-upgrade backup is no longer a restore point once operations resume. Take a fresh one and verify it.
A note on rollback
Rollback becomes progressively less real as the window proceeds. Restoring a database is straightforward. Restoring a database after controllers have been updated to a firmware level the old head end does not support is not, because the controllers now have to come back too. That is the point at which a two-hour rollback becomes an all-night one.
Decide the abort criteria before starting, write down the latest time at which rollback is still viable, and put a name against the decision. In practice the useful discipline is to sequence the change so the least reversible steps happen last, and to complete verification of each reversible stage before beginning the next.
Skyfal Security provides programming, configuration and upgrade support for these platforms as an independent service company. It is not an authorized dealer, reseller or partner of any manufacturer, and does not sell licences. See Programming and Configuration and Systems We Support.
Plan the window from evidence, not from an estimate
A rehearsal upgrade on a restored copy gives you the real duration and the real defect list before anything is scheduled.