Air-Gap Operational Runbook

Operational procedures complementary to the Air-Gap Deployment guide. Where the deployment guide explains the technical architecture, this runbook covers the day-to-day operational concerns: who handles which disc, how often keys rotate, what to do when transport goes wrong, and how to respond when a signature verification fails.

Chain of custody

Every air-gap disc is a privileged-access artifact: it carries the operator's ed25519 signature and, on the public→private direction, the union of every package the air-gapped fleet will install on its next update cycle. Treat each disc the same way you would treat a signed-installer USB key for production hosts.

Label every disc immediately at burn time with: collector hostname, ISO label, run timestamp, and disc N of M (for multi-disc sets). The run timestamp matches what's in the manifest, so a misplaced disc can be cross-referenced against the collector's audit log.
Maintain a transport log (paper or air-gapped spreadsheet) listing: who carried each disc set, when it left the public-side facility, when it was inserted at the private-side facility, and the SHA-256 of each ISO at both ends.
Store unburned ISO masters on the collector for at least the retention window of one complete delta-collection cycle (default 30 days) so a lost-in-transit disc set can be re-burned without redoing the mirror sync.
Destroy obsolete discs (older than two retention windows) by physical shredding rather than deletion — even ingested data carries the operator's signature, which has cryptographic value beyond the package contents themselves.

Ed25519 keypair rotation

The collector signs every manifest with an ed25519 private key; the repository verifies against the corresponding public key stored in its trusted-collector keyring directory (/var/lib/sysmanage/airgap/trusted-collectors/). The keypair is generated automatically the first time you set the server role to collector (no manual keygen step). Recommended cadence:

Annual rotation baseline. Generate a new keypair on the collector, distribute the new public key to the repository (over an existing trusted channel — sneakernet, signed email, or the previous trusted ingest), and run one transition disc that's signed by both keys before retiring the old key.
Immediate rotation on suspected compromise. Symptoms include: a stolen / lost laptop that holds the collector's private key, an unauthorized signature appearing on a disc the operator didn't burn, or any indication the collector host was compromised. Rotate keys, and treat every disc signed by the old key as untrusted (re-collect + re-burn before ingesting on any repository).
Private-key storage. The collector's private key (/var/lib/sysmanage/airgap/collector-ed25519.pem, generated automatically on role selection, mode 0600) should be readable only by the SysManage server process and never copied off the collector. The ensure_collector_keypair step never overwrites an existing key, so re-selecting the collector role is safe. Backup via a sealed-envelope hardware token (YubiKey, Nitrokey) stored in a different physical location from the collector itself — the same threat model as your CA's signing key.
Public-key distribution. The repository's keyring directory can hold multiple trusted public keys during a rotation window — each as a separate .pub file. New ingestions verify against any key in the directory; remove a retired key only after the last disc signed by it has been ingested. To rotate, drop the new collector's .pub into the keyring (additive — existing keys are never cleared).

Automated key provisioning & ingestion

Recent releases automate the key-distribution and ingestion steps that the rotation procedures above describe manually. Understanding the automated path tells you what to check when something doesn't work:

Keygen on role selection. Setting the server role to collector in Settings → Server Role mints the ed25519 keypair at the canonical path. The role lives in the database (not sysmanage.yaml), so a fresh server boots as standard and the key is created the moment you choose the collector role.
Public key rides the install bundle. When the Pro+ air-gap bundle is built on the collector (scripts/buildAirGapBundle.sh server), it embeds the collector's .pub into the bundle. The repository's overlay installer drops it into /var/lib/sysmanage/airgap/trusted-collectors/ automatically — so a repository provisioned from that bundle trusts the collector with no separate key exchange. Only the public key travels; the private signing key never leaves the collector.
Strict verification by default. The repository rejects unsigned and HMAC-fallback manifests unless airgap.verify_strict is set to false in sysmanage.yaml. Leave it strict in production; loosen it only for a dev loop where cryptography isn't installed on the collector.
Ingestion is driven, not manual. POSTing an ISO path to the repository's ingest endpoint inserts a QUEUED AirgapIngestionRun; a background orchestrator then loop-mounts the ISO read-only, verifies the embedded /manifest.json signature against the keyring, rsyncs the verified payload into /var/lib/sysmanage/airgap-repo, and registers a per-distro local repository. Watch a stuck run's status: VERIFYING_SIG means the signature step is running or failed (check the keyring), COPYING means verification passed and the rsync is in flight.

Transport-loss procedures

When a disc set fails to arrive at the air-gapped facility within its expected transit window:

Verify with the courier or carrier that the package was actually dispatched. Most "lost" discs are actually misrouted; reconcile the transport-log entry against tracking data.
Decide between waiting and re-burning based on operational urgency. If the air-gapped fleet has a fresh-enough mirror (compliance UI shows current or stale, not very_stale), waiting one additional courier cycle is acceptable.
If re-burning: locate the matching ISO master in the collector's /var/lib/sysmanage/airgap-iso/ retention window, re-burn from the same source ISO so the SHA-256 matches the lost disc's transport-log entry. Do not regenerate the manifest — that would make the new disc look like a separate run to the repository's audit trail.
If the original disc arrives after a re-burn has been ingested, log both ingestions in the transport log; the second ingest will succeed (idempotent — same content) and creates an audit-trail crumb that helps reconcile the next cycle's gap analysis.
Physically destroy any disc whose chain-of-custody is broken (carrier handover skipped, package found unsealed, etc.) regardless of whether the contents seem intact. The cost of re-burning is small; the cost of ingesting tampered media on the air-gapped network is large.

Signature verification failure

When the repository's ingestion endpoint reports signature does not match manifest payload, treat it as a security incident until proven otherwise:

Do not retry ingestion. Repeated signature-verification failures don't make the data more trustworthy. Set the disc aside in a tagged "pending-investigation" envelope.
Verify the collector's public key is present in the repository keyring. The most common (benign) cause is a key rotation that didn't propagate fully. Re-export the public key from the collector (/var/lib/sysmanage/airgap/collector-ed25519.pub), compare to the .pub files in the repository's /var/lib/sysmanage/airgap/trusted-collectors/ keyring, drop in the missing key if absent, retry ingestion.
If keys match, inspect the disc. Tampering would typically also break the per-file SHA-256 verification, so a signature-only failure on a disc whose hashes pass is suspicious. Inspect the disc surface for physical damage; mount the ISO read-only and verify file checksums against the manifest manually.
If the disc cannot be explained by key drift or physical damage, escalate. Quarantine the disc, file an incident with your security team, and rotate the collector's keypair as a precaution. The next collection cycle should be performed on a freshly-imaged collector if the original collector itself is suspect.

Recommended cadences

Defaults that work for most deployments — adjust against your organization's tolerance for delayed patch availability:

Collection cycle: weekly, with a delta-mode top-up daily. The weekly full-snapshot ensures the cumulative disc set stays current; the daily delta keeps the air-gapped network within one business day of upstream security advisories.
Transport: match collection cadence. A disc that sits in transit longer than the next collection cycle is wasteful — schedule courier pickups for the same day as the burn step.
Ingestion: on arrival. Don't queue media; the repository's freshness label drifts toward stale the moment a disc is sitting on the operator's desk instead of being ingested.
Key rotation: annual baseline; immediate on compromise. Calendar a key-rotation event for the same date every year so it doesn't get forgotten.
Audit review: quarterly. Reconcile the transport log against the collector's AirgapCollectionRun table and the repository's AirgapIngestionRun table. Each pair should match by signer_fingerprint + collector_iso_label; discrepancies surface ingestion gaps before they grow into compliance failures.