openDesk Hilfe

Developing openDesk deployment automation

Active development on the deployment is currently only available to project members. However, contributions are possible using the CLA process.

Overview

The following sketch provides a high-level overview to get a basic understanding of the deployment-relevant structure of this repository. Understanding that structure is vital if you want to contribute to developing the openDesk platform.

The helmfile.yaml.gotmpl file in the root folder is the foundation for the entire deployment. It references the helmfile_generic.yaml.gotmpl file, which includes app-specific helmfile.yaml.gotmpl files and global values files located in ./environments/default.

helmfile.yaml.gotmpl also refers to three predefined environments: dev, test, and prod.

The helmfile_generic.yaml.gotmpl file is designed to be referenced from external repositories, where custom environments may be defined. An example is demonstrated in the helmfile.yaml.gotmpl file.

Before you investigate any app-specific configuration, it is recommended that you review the contents of ./helmfile/environments/default to understand what configuration details are set there, as the app deployments reference them.

Default branch, develop and other branches

The main branch is configured to be the default branch, as visitors to the project on openCode should see that branch by default.

Please use the develop branch to diverge your branch(es) from. See the workflow guide for more details on naming conventions.

External artifacts - charts.yaml.gotmpl and images.yaml.gotmpl

The charts.yaml.gotmpl and images.yaml.gotmpl files are the central place to reference any external artifacts used for the deployment.

Besides the deployment automation itself, some tools work with the contents of the files:

  • Linting: Ensures consistency of the file contents for the other tools.
  • Renovate: Automatically create MRs that update the components to their latest version.
  • Mirror: Mirror artifacts to openCode.

Please find details on these tools below.

Linting

In the project’s CI, there is a step dedicated to lint the two yaml files, as we want them to be in

  • alphabetical order regarding the components
  • logical order regarding the non-commented lines (registry > repository > tag).

In the linting step, the openDesk CI CLI is used to apply the aforementioned sorting, and the result is compared with the unsorted version. If there is a delta, the linting fails, and you can fix it by running the CLI tool locally, verifying and applying the result to your branch.

Note

Please ensure that in component blocks, you use comments only at the beginning of the block or at its end. Ideally, you stick with the many available examples in the yaml files.

Example:

 synapse:
 # providerCategory: "Supplier"
 # providerResponsible: "Element"
 # upstreamRegistry: "https://registry-1.docker.io"
 # upstreamRepository: "matrixdotorg/synapse"
 # upstreamMirrorTagFilterRegEx: '^v(\d+)\.(\d+)\.(\d+)$'
 # upstreamMirrorStartFrom: ["1", "91", "2"]
 registry: "registry.opencode.de"
 repository: "BMI/opendesk/components/supplier/element/images-mirror/synapse"
 tag: "v1.91.2@sha256:1d19508db417bb2b911c8e086bd3dc3b719ee75c6f6194d58af59b4c32b11322"

Disable linting selectively

If you follow the “push early, push often” paradigm to save your work to the central Git instance, or you just fix a typo in existing documentation, you can avoid the CI and its linting being executed, as it might not offer additional value.

GitLab offers two options to skip the CI on a commit/push:

  1. Add [ci skip] to your commit message (details). Note: The string has to be removed before merging your feature branch into develop.
  2. Use the related git push option git push -o ci.skip (details).

Renovate

Uses a regular expression to match the values of the following attributes:

  • # upstreamRegistry required: Attribute’s value must be prefixed with https:// for Renovate.
  • # upstreamRepository required
  • tag required

Checks for newer versions of the given artifact and creates an MR containing the newest version’s tag (and digest).

Mirroring

openDesk strives to make all relevant artifacts available on openCode so there is a mirroring process configured to pull artifacts that do not originate from openCode into projects called *-Mirror within the openDesk Components section.

The mirror follows a release-oriented strategy: It mirrors exactly the artifact versions (tag and digest) pinned in the charts.yaml.gotmpl and images.yaml.gotmpl files of the environments default and default-enterprise-overrides into the target repositories referenced in those files (registry/repository/name). Related artifacts of the pinned version (SBOM, VEX, attestations and signatures) are mirrored as well and refreshed when they change at the source.

The mirror script takes the information on where to pull an artifact from the annotations inside the two yaml files:

  • # upstreamRegistry required: To identify the source registry. Sources already located on openCode are skipped.
  • # upstreamRepository required: To identify the source repository

Enterprise components that are pulled from a supplier’s non-public registry use the following annotations in addition. Their values name environment variables that are configured in the mirror’s CI:

  • # enterpriseRegistryBase: Prefix of the environment variables holding the source registry connection details:
    • <enterpriseRegistryBase>_URL
    • <enterpriseRegistryBase>_USERNAME
    • <enterpriseRegistryBase>_PASSWORD
  • # enterpriseRegistryPath: Name of the environment variable holding the source repository path within that registry.

The tag-range annotations are evaluated to log newer versions as potential mirror candidates - they are no longer mirrored automatically:

  • # upstreamMirrorTagFilterRegEx optional: Only tags that match the given regular expression and are newer than the pinned version are listed as candidates. Note: You must use single quotes for this attribute’s value if you use backslash leading regex notation like \d.
  • # upstreamMirrorStartFrom optional: Array of numeric values defining the minimum version for candidate listing in case the pinned version does not match the filter regex. You must use capturing group in # upstreamMirrorTagFilterRegEx to identify the single numeric elements of the version within the tag and use per capturing group (left to right) one numeric array element here to define the version the candidate listing should start with.

Find more details and the code of the mirror script here: https://gitlab.opencode.de/bmi/opendesk/tooling/oci-pull-mirror

Note

The mirror is scheduled to run every hour at 42 minutes past the hour.

Get new artifacts mirrored

Every scheduled run mirrors the versions pinned in the develop branch, so once your branch is merged into develop, related artifacts are updated automatically.

If you need the artifacts pinned in your branch mirrored before it is merged, open an issue in the oci-pull-mirror project with your branch name as the issue’s subject. The next scheduled run additionally processes every branch named by an open issue and closes the issue with a summary report once the branch was mirrored successfully; on failures the issue stays open and and will be retried on the next run.

Note

The mirror does not create target repository structures: If the target project does not exist yet, it has to be created before the mirror can push to it.

Creating new charts/images

When you create new Helm charts, please check out the openDesk Best Practices for Helm charts.

You may also want to make use of our standard CI to quickly get Helm charts and container images that are signed, linted, scanned, and released. Check out the .gitlab-ci.yaml files in the project’s Charts or Images to get an idea just how little you need to do by yourself.