Air-Gapped Operation

What “air-gapped” means

An air-gapped system is one that is physically or logically isolated from any external network, most commonly from the Internet. The term originates from the literal air gap between two networks: no wire, radio or network route connects the isolated system to the outside world.

The isolation is a security measure. Because there is no path into the system, remote attacks cannot reach it, and data cannot silently leak out over the network. Public-sector and classified environments often enforce such isolation because the data they handle must not leave the protected perimeter, or because the system must keep working even if the surrounding network is compromised.

Nothing is pulled from the Internet at runtime. openDesk’s software is brought in deliberately, through a controlled channel. Everything that normally arrives over the network must be transported manually.

Current state of things

Updated for openDesk 1.19.

• Air-gapped operation is experimental, and upstream does not test it yet. openDesk Community Edition (CE) and Enterprise Edition (EE) may run in air-gapped environments. Air-gapped operation is not yet a supported test case upstream, and we do not officially support it yet. Some components may still make outbound calls. Right now we are not aware of a case where this prevents operation.
• Scope. openDesk currently optimises for air-gapped installation and operation. Third-party applications — for example the provisioning of mobile apps or signalling — are not part of our tests and may still require external services.
• openDesk delivers everything itself. openDesk ships all of its own components: the container images, the deployment repository and the Helm charts for one release come as a single collection of artifacts. Nothing else has to be fetched from the Internet.
• External services ship as native services. The all-in-one installation provides the services that would otherwise be external as native services in the cluster. Nothing is missing for normal operation. See External Services. At high scaling, however, operation can still fail.

Recommended Excursus: on-premises and air-gapped operation

On-premises operation means running openDesk in your own infrastructure or data centre instead of a public cloud: you control the hardware, the network and the data location. Air-gapped operation is the strongest form of it — full and permanent isolation from every external network.

The two are not alternatives. They are two points on one scale, and that scale is availability. On-premises operation with Internet access is useful and completely normal, as long as the uplink works. But an on-premises deployment cannot guarantee an external connection: outages, restricted capacity or a link that only recovers slowly are not under your control, and every uplink is a dependency on a third party.

Use the uplink while it works — but design so that openDesk never depends on it. From that follows the rule:

The higher the availability requirement for an on-premises deployment, the fewer external dependencies it may have.

At the top of that scale stands air-gapped operation: no external dependency at all. Everything openDesk needs is available locally, so a lost or degraded external connection is an event you absorb, not one that stops operation.

What you have to do

To run openDesk offline, you must bring in everything it needs before it starts and keep that stock updated over its lifetime. For the Kubernetes-based stack, that means at least these tasks:

1. Fetch the images. Every container image openDesk uses must be fetched on the DMZ side, then carried to the isolated system and loaded into a local registry.
2. Set up a local registry. The container runtime inside the isolated system must pull exclusively from this registry — not from the public registries that are unreachable there. Reference systems are tested with zot as a pull-through mirror.
3. Bring in the deployment repository, the charts and the renderer. The openDesk deployment repository holds the helmfiles and the chart references; the Helm charts it references must be available as an offline cache. Rendering the deployment needs its own tooling: helm, helmfile, the Helm plugins they rely on — openDesk uses Helm Diff — and yq. See Requirements for the supported versions. The repository, the charts and the tooling must all exist on the isolated system, and the repository is mandatory — without it openDesk cannot be deployed. The platform-development group packages the tooling as an all-in-one image, which the product tests use: helm.
4. Add the certificates. TLS certificates cannot be issued by an ACME challenge, because the challenge needs the domain to be reachable. Obtain them beforehand and apply them as static secrets.
5. Deploy from local sources only. The deployment must run entirely from the local registry, the deployment repository, the chart cache and the local tooling, with no network access.

The recurring pattern is the same for every artifact: fetch on the DMZ side → carry across the air gap → ingest into the isolated system.

Certificates across the gap

Certificates need separate handling. In most organisations they come from an internal PKI rather than a public certificate authority; a certificate from a public CA (an external certificate) is possible too, but for an isolated system the usual case is a certificate issued by the organisation’s own PKI. Either way an ACME challenge (Let’s Encrypt, cert-manager) cannot be used: it needs the domain to be reachable from the Internet, and inside the isolation it is not.

For an isolated system, the certificate is fetched before the gap, on the DMZ side, and applied as a static secret inside:

1. Obtain: get the certificate before the gap — usually by requesting it from your internal PKI (or from an internal ACME endpoint the DMZ can reach). If you use a public CA instead, a DNS-based challenge (DNS-01) is the method that does not need the domain to be reachable. A wildcard certificate for the whole domain is typical, so a single artifact covers every service.
2. Carry: bring the certificate files (fullchain.pem, private.key) across the gap like any other artifact.
3. Apply: install the certificate as a static secret on the isolated system — no challenge involved.

On the isolated system the certificate becomes a kubernetes.io/tls secret. A helmfile environment override tells the deployment to use that secret instead of cert-manager: it sets apps.certificates.enabled: false and certificate.create: false.

An internal certificate is not part of the public CA chains, so clients and cluster components do not trust it by default. openDesk can still use it: it can set up the matching trust chain by bundling your CA into the components that need it. The trust settings control this — trust.create, trust.certificateAuthorities (a plain-text certificate or an existing secret) and trust.secret — and some components additionally need a keystore and a truststore. See Self-signed certificates and custom CA — Trust for the details.

Local registry

Every image pull inside the isolated system must go through a local registry, and the container runtime must be configured so that it never falls back to the Internet: a pull either resolves locally or fails.

Reference systems are tested with zot, a pull-through mirror that speaks plain HTTP. The mirror keeps the upstream path and the original digests, so the digest-pinned openDesk deployment stays valid, and containerd’s override_path resolves every image through that path. The container runtime is pointed at the mirror with a certs.d configuration; the image references in the deployment stay unchanged.

A working example — the zot deployment with its configuration, plus the containerd wiring — ships with the experimental CLI: scripts/zot.

The opendesk-airgap-cli

opendesk-airgap-cli is a tool for this workflow. It covers the five tasks above on both sides of the air gap and runs as a stateless Docker image that bundles skopeo, crane, helm, helmfile, kubectl, git, yq and openssl. The artifact lives in a directory that you mount as a volume — that directory is the only thing that crosses the air gap.

How the tool is used — its sub-commands, flags, the artifact layout and the Docker invocation — is documented in the tool’s own README.