Repository Mirroring

Overview

The repository_mirroring_engine Pro+ module emits apply_deployment_plan messages that the agent on the mirror host runs. The OSS server owns the configuration tables, the Settings UI, and the routes that 402 without the engine loaded; the engine owns plan generation. No agent code needs to change to support a new package manager — every plan goes through the existing apply_deployment_plan handler.

Supported Package Managers

• APT (Debian / Ubuntu) — uses apt-mirror with optional GPG signing-key fetch.
• DNF / YUM (RHEL / Oracle Linux / Fedora / Rocky / Alma) — uses dnf reposync --newest-only --delete + createrepo_c --update.
• zypper (openSUSE / SLES) — uses a portable zypper download '*' wrapper + createrepo_c.
• pkg (FreeBSD) — uses pkg fetch -Loy -a + pkg repo.

Per-Mirror Configuration

Each mirror is an admin-managed row in the mirror_repository table. Common fields:

• name — short slug, used as the on-disk subdirectory name. Path-traversal characters are rejected.
• upstream_url — the public mirror you sync from.
• sync_cron — POSIX cron (5 fields). Defaults to 0 4 * * * (4 a.m. local). Evaluated by the same parser as upgrade profiles in automation_engine.
• bandwidth_cap_kbps — when non-zero, every fetch is wrapped with trickle -d <cap> if available; falls through uncapped otherwise. Most ops teams cap at the network layer (tc); this is a per-job courtesy.
• network_tier — free-form tag (e.g. us-east, edge) that lets clients pick the closest mirror.
• host_id — the agent that runs the sync plan. The on-disk tree lives at {mirror_root_path}/{name} on that host.

Per-package-manager fields: APT uses suite + components + optional signing_key_url; DNF uses repoid + optional gpgkey_url; zypper uses repo_alias; pkg uses optional release (e.g. FreeBSD:14:amd64).

Snapshots & Restore

A snapshot is an rsync -aH --delete of the live tree into a sibling read-only directory at {mirror_root}/{name}/.snapshots/{snapshot_id}. Snapshot IDs are sortable timestamps (YYYYMMDDTHHMMSS). After the rsync the engine runs chmod -R a-w so the snapshot can't be modified in place.

Restore is the reverse — an rsync from the chosen snapshot back into the live tree. The engine doesn't symlink-flip, because most mirror servers (httpd, nginx) cache-on-stat and the in-place rsync is simpler to reason about than coordinating a flip across worker processes.

Snapshot retention is governed by snapshot_count_to_keep. The garbage-collection plan lists snapshots in alphabetical (= chronological) order, drops everything older than the keep count, and rm -rfs the rest after relaxing the read-only flag.

Air-Gapped Deployments

The mirroring engine produces the public-side ("collector") tree that feeds optical media or removable storage into an air-gapped network. The private-side ("repository") host imports the media into its own mirror tree using the same engine — the import is just an offline rsync of the snapshot directory. From there, hosts on the private network treat the repository host as their normal upstream.

Admin Settings

The singleton mirror_settings row carries fleet-wide defaults:

• mirror_root_path — filesystem root that every mirror's subdirectory hangs off (default /var/mirror).
• integrity_check_cadence_hours — how often the integrity-check plan runs against each mirror (default 24).
• retention_window_days — days a snapshot stays on disk before being eligible for GC (default 30).
• default_bandwidth_cap_kbps — applied when a per-mirror cap is not set (default 0 = uncapped).
• snapshot_count_to_keep — number of snapshots the GC plan preserves (default 10).

API Reference

Every route below returns 402 without repository_mirroring_engine loaded.

• GET /api/mirror-repositories — list mirrors
• POST /api/mirror-repositories — create a mirror
• GET /api/mirror-repositories/{id} — read one mirror
• PUT /api/mirror-repositories/{id} — update a mirror
• DELETE /api/mirror-repositories/{id} — delete a mirror
• POST /api/mirror-repositories/{id}/sync — fire the sync plan immediately
• POST /api/mirror-repositories/{id}/snapshot — take a snapshot now
• POST /api/mirror-repositories/{id}/restore/{snapshot_id} — restore a snapshot
• GET /api/mirror-repositories/{id}/snapshots — list snapshots
• POST /api/mirror-repositories/tick — cron-driven driver hook
• GET /api/settings/mirror / PUT — admin settings (legacy singleton)
• GET /api/mirror-platform-configs — list per-platform configs
• POST /api/mirror-platform-configs — create a platform config
• PUT /api/mirror-platform-configs/{id} — update a platform config
• DELETE /api/mirror-platform-configs/{id} — delete a platform config (refuses while it owns mirrors)
• GET /api/mirror-repositories/setup-status/{host_id} — read tooling setup status for a host
• POST /api/mirror-repositories/setup-status/{host_id}/refresh — queue a tooling probe (asynchronous)
• POST /api/mirror-repositories/setup-install/{host_id} — queue an install of the tooling for a package manager

Per-platform tab strip (Phase 10.4.3)

The Settings → Repository Mirroring page is laid out as a tab strip with one tab per package manager. Today the four tabs are Ubuntu/Debian (apt), RHEL/Fedora (dnf — covers RHEL, Fedora, Oracle Linux, Rocky, Alma, CentOS Stream), openSUSE/SLES (zypper), and FreeBSD (pkg). Each tab represents one mirror host running one PM, so the Add Mirror dialog inherits its package manager from the active tab and the host picker filters to OS-family matches (Linux hosts on apt/dnf/zypper tabs, FreeBSD hosts on the pkg tab).

Snaps: not supported. Mirroring snap packages requires Canonical's commercial Snap Store Proxy product (separate auth + HTTP API, not file-level), which is out of scope for this engine. OpenBSD, NetBSD, macOS, and Windows are deferred until their own tooling probes and install plans are written.

Each tab contains three cards stacked vertically:

• Platform Settings — searchable host picker, mirror root path on the host, retention window, integrity-check cadence, default bandwidth cap, and snapshot keep count. These values are scoped to this platform on this host, so a Linux mirror on one host and a FreeBSD mirror on another can have completely independent storage layouts.
• Mirror Setup Status — chips for each tool the engine's plan-builders shell out to (apt-mirror, reposync, createrepo_c, trickle, rsync, curl, etc.). Refresh button queues a probe; Install Tools button queues a sudo install of the platform-appropriate package set. Both go through the agent's standard apply_deployment_plan path; the card polls the cached status row until the agent's command_result lands.
• Mirror Repositories — the table of upstream URLs being mirrored on this platform, plus inline sync/snapshot/edit/delete actions and an Add Mirror dialog. The package manager dropdown in the dialog is filtered to those the active platform supports (apt/dnf/zypper for Linux, pkg for FreeBSD).

If the active tab has no platform configuration yet, an empty-state card prompts you to pick a host. The host picker only lists hosts whose detected platform matches the tab — Linux hosts on the Linux tab, FreeBSD hosts on the FreeBSD tab — to prevent misconfigured pairings.

Version dropdown + known-version catalog (Phase 10.4.4)

The Add Mirror dialog's version field is sourced from a pre-populated mirror_known_version catalog instead of free-text. Selecting a row auto-fills the upstream URL and the per-PM identifier (suite for apt, repoid for dnf, repo_alias for zypper, release for pkg) so an operator can't fat-finger noblee and silently produce a broken mirror. The catalog ships seeded with the canonical Ubuntu/Debian/RHEL-family/openSUSE/FreeBSD versions; future versions land via dedicated migrations rather than auto-discovery, keeping the supported set reviewable in code.

One physical host can mirror multiple OS families that share a package manager. RHEL/Rocky/Alma/Fedora all use dnf with different upstream URLs, so they appear as separate mirror_repository rows under the same RHEL/Fedora tab — each row carries its own known_version_id referencing a different catalog entry. The Default Package Mirrors card has independent rows for each, so an admin can assign different defaults to RHEL 9 hosts vs Rocky 9 hosts even when both pull from the same physical mirror server.

Default Package Mirrors (Phase 10.4.4)

Settings → Host Defaults gains a "Default Package Mirrors" card that drives an apply/revert workflow against active hosts. One row per (platform, version_key, os_family) tuple drawn from the catalog; each row's dropdown lists the eligible mirrors (right PM + at least one successful sync) plus a "Cloud (upstream default)" option.

Saving a non-cloud choice queues an apply plan to every active host whose platform_release matches the catalog row's regex — simultaneous rollout, no staggered windows. Choosing "Cloud" queues a revert plan to the same matching hosts. New host registrations and approvals invoke the same hook automatically, so a freshly-enrolled host of a covered family gets pointed at the mirror without operator action.

The API hard-blocks (HTTP 409) any attempt to assign a mirror that hasn't completed a successful sync. This prevents pointing live clients at an empty or partially-built mirror tree where apt update would fail. Re-attempt once the mirror's last_sync_status reaches SUCCESS.

The apply mechanism is additive only. Each PM gets a single override file the agent drops into a directory the package manager scans alongside operator-edited config:

• apt: /etc/apt/sources.list.d/zzz-sysmanage-mirror.list + apt-get update
• dnf: /etc/yum.repos.d/zzz-sysmanage-<repoid>.repo with the same [section] header as the upstream — dnf reads files in lex order and the last definition wins, so our zzz- prefix overrides whatever shipped + dnf clean all
• zypper: /etc/zypp/repos.d/zzz-sysmanage-<alias>.repo + zypper refresh
• pkg: /usr/local/etc/pkg/repos/sysmanage-mirror.conf overriding the FreeBSD repo + pkg update -f

Revert is symmetric: each plan deletes the override file and refreshes the package manager's metadata cache. Operator-edited config (the original sources.list, vendor-shipped .repo files) is never modified, so a manual edit between apply and revert is preserved.

API surface: GET /api/mirror-known-versions for the catalog dropdown, GET /api/host-defaults/mirrors for the assignment table, PUT /api/host-defaults/mirrors/{platform}/{version_key}/{os_family} with {mirror_id} body to assign or {mirror_id: null} to revert to cloud.