GitLab CI template for Kubernetes (k8s)

This project implements a generic Gitlab CI template for Kubernetes environments.

Overview

This template implements continuous delivery/continuous deployment for projects hosted on Kubernetes platforms.

It provides several features, usable in different modes.

Review environments

The template supports review environments: those are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment. It is only active for non-production, non-integration branches.

It is a strict equivalent of GitLab's Review Apps feature.

It also comes with a cleanup job (accessible either from the environments page, or from the pipeline view).

Integration environment

If you're using a Git Workflow with an integration branch (such as Gitflow), the template supports an integration environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment. It is only active for your integration branch (develop by default).

Production environments

Lastly, the template supports 2 environments associated to your production branch (master by default):

• a staging environment (an iso-prod environment meant for testing and validation purpose),
• the production environment.

You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

• continuous deployment: automatic deployment to production (when the upstream pipeline is successful),
• continuous delivery: deployment to production can be triggered manually (when the upstream pipeline is successful).

Usage

Include

In order to include this template in your project, add the following to your gitlab-ci.yml:

include:
  - project: 'to-be-continuous/kubernetes'
    ref: '2.0.2'
    file: '/templates/gitlab-ci-k8s.yml'

Supported cluster access methods

The Kubernetes template supports 3 ways of login/accessing your Kubernetes cluster(s):

1. The GitLab Kubernetes integration: when enabled, the template automatically retrieves and uses your Kubernetes cluster configuration (KUBECONFIG env),
2. By defining an explicit kubeconfig yaml content from env,
3. By defining explicit kubeconfig exploded parameters from env: server url, server certificate authority and user token.

Global configuration

The Kubernetes template uses some global configuration used throughout all jobs.

Secrets management

Here are some advices about your secrets (variables marked with a 🔒):

1. Manage them as project or group CI/CD variables:
   • masked to prevent them from being inadvertently displayed in your job logs,
   • protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).
2. In case a secret contains characters that prevent it from being masked, simply define its value as the Base64 encoded value prefixed with @b64@: it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: $ -> $$).

Environments configuration

As seen above, the Kubernetes template may support up to 4 environments (review, integration, staging and production).

Each deployment job produces output variables that are propagated to downstream jobs (using dotenv artifacts):

• environment_type: set to the type of environment (review, integration, staging or production),
• environment_name: the application name (see below),
• environment_url: set to $CI_ENVIRONMENT_URL.

They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

Here are configuration details for each environment.

Review environments

Review environments are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

They are disabled by default and can be enabled by setting the K8S_REVIEW_SPACE variable (see below).

Here are variables supported to configure review environments:

Note: By default review environment.url will be built as ${K8S_REVIEW_ENVIRONMENT_SCHEME}://${$CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${K8S_REVIEW_ENVIRONMENT_DOMAIN}

Integration environment

The integration environment is the environment associated to your integration branch (develop by default).

It is disabled by default and can be enabled by setting the K8S_INTEG_SPACE variable (see below).

Here are variables supported to configure the integration environment:

Staging environment

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (master by default).

It is disabled by default and can be enabled by setting the K8S_STAGING_SPACE variable (see below).

Here are variables supported to configure the staging environment:

Production environment

The production environment is the final deployment environment associated with your production branch (master by default).

It is disabled by default and can be enabled by setting the K8S_PROD_SPACE variable (see below).

Here are variables supported to configure the production environment:

Deployment jobs

The GitLab CI template for Kubernetes supports two policies for deploying your code:

1. script-based deployment
2. template-based deployment

1: script-based deployment

In this mode, you only have to provide a shell script that fully implements the deployment using the kubectl CLI.

The deployment script is searched as follows:

1. look for a specific k8s-deploy-$env.sh in the $K8S_SCRIPTS_DIR directory in your project (e.g. k8s-deploy-staging.sh for staging environment),
2. if not found: look for a default k8s-deploy.sh in the $K8S_SCRIPTS_DIR directory in your project,
3. if not found: the GitLab CI template assumes you're using the template-based deployment policy.

Your script(s) may use dynamic variables.

2: template-based deployment

In this mode, you have to provide a Kubernetes deployment file in your project structure, and let the GitLab CI template kubectl apply it.

The template processes the following steps:

1. optionally executes the k8s-pre-apply.sh script in your project to perform specific environment pre-initialization (for e.g. create required services),
2. looks for your Kubernetes deployment file, substitutes dynamic variables and kubectl apply it,
   1. look for a specific deployment-$env.yml in your project (e.g. deployment-staging.yml for staging environment),
   2. fallbacks to default deployment.yml.
3. optionally executes the k8s-post-apply.sh script in your project to perform specific environment post-initialization stuff,
4. optionally executes the k8s-readiness-check to wait & check for the application to be ready (if not found, the template assumes the application was successfully started).

All scripts and Kubernetes deployment files may use dynamic variables.

Cleanup jobs

The GitLab CI template for Kubernetes supports two policies for destroying an environment (actually only review environments):

1. script-based cleanup
2. template-based cleanup

1: script-based cleanup

In this mode, you only have to provide a shell script that fully implements the environment cleanup using the kubectl CLI.

The a deployment script is searched as follows:

1. look for a specific k8s-cleanup-$env.sh in the $K8S_SCRIPTS_DIR directory in your project (e.g. k8s-cleanup-staging.sh for staging environment),
2. if not found: look for a default k8s-cleanup.sh in the $K8S_SCRIPTS_DIR directory in your project,
3. if not found: the GitLab CI template assumes you're using the template-based cleanup policy.

Your script(s) may use dynamic variables.

TIP: a nice way to implement environment cleanup is to declare the label app=${appname} on every Kubernetes object associated to your environment. Then environment cleanup can be implemented very easily with command kubectl delete all,pvc,secret,ingress -l "app=${appname}"

2: template-based cleanup

In this mode, you mainly let Kubernetes delete all objects from your Kubernetes deployment file.

The template processes the following steps:

1. optionally executes the k8s-pre-cleanup.sh script in your project to perform specific environment pre-cleanup stuff,
2. looks for your Kubernetes deployment file, substitutes dynamic variables and kubectl delete it,
   1. look for a specific deployment-$env.yml in your project (e.g. deployment-staging.yml for staging environment),
   2. fallbacks to default deployment.yml.
3. optionally executes the k8s-post-cleanup.sh script in your project to perform specific environment post-cleanup (for e.g. delete bound services).

All scripts and Kubernetes deployment files may use dynamic variables.

Cleanup job limitations

When using this template, you have to be aware of one limitation (bug) with the cleanup job.

By default, the cleanup job triggered automatically on branch deletion will fail due to not being able to fetch the Git branch prior to executing the job (obviously because the branch was just deleted). This is pretty annoying, but as you may see above, deleting an env may require scripts from the project...

In a future version, the template will rely on Kustomize and will be able to delete an entire environment using the app label. In the meantime we're just sorry about this bug, but for now there is not much we can do:

• remind to delete your review env manually before deleting the branch
• otherwise you'll have to do it afterwards from your computer (using kubectl CLI) or from the some k8s web console.

Dynamic Variables

You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments (review, integration, staging and production), each with different application names, exposed routes, settings, ...

Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).

In order to be able to implement some genericity in your scripts and templates, you should use available environment variables:

1. any GitLab CI variable (ex: ${CI_ENVIRONMENT_URL} to retrieve the actual environment exposed route)
2. any custom variable (ex: ${SECRET_TOKEN} that you have set in your project CI/CD variables)
3. dynamic variables set by the template:
   • ${appname}: the application target name to use in this environment (ex: myproject-review-fix-bug-12 or myproject-staging)
   • ${appname_ssc}: the application target name in SCREAMING_SNAKE_CASE format (ex: MYPROJECT_REVIEW_FIX_BUG_12 or MYPROJECT_STAGING)
   • ${env}: the environment type (review, integration, staging or production)
   • ${hostname}: the environment hostname, extracted from ${CI_ENVIRONMENT_URL} (has to be explicitly declared as environment:url in your .gitlab-ci.yml file)
   • any GitLab CI variable (ex: ${CI_ENVIRONMENT_URL} to retrieve the actual environment exposed route)

⚠️

In order to be properly replaced, variables in your YAML descriptors shall be written with curly braces (ex: ${MYVAR} and not $MYVAR).

Multiline variables must be surrounded by double quotes ("). Example:

[...]
containers:
- name: restaurant-app
  env:
  # multiline variable
  - name: MENU
    value: "${APP_MENU}"

kube-score job

The Kubernetes template enables kube-score analysis of your Kubernetes object definitions. This job is mapped to the test stage and is active by default.

Here are its parameters:

Variants

Vault variant

This variant allows delegating your secrets management to a Vault server.

Configuration

In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters:

Usage

Then you may retrieve any of your secret(s) from Vault using the following syntax:

@url@http://vault-secrets-provider/api/secrets/{secret_path}?field={field}

With:

Example

include:
  # main template
  - project: 'to-be-continuous/kubernetes'
    ref: '2.0.2'
    file: '/templates/gitlab-ci-k8s.yml'
  # Vault variant
  - project: 'to-be-continuous/kubernetes'
    ref: '2.0.2'
    file: '/templates/gitlab-ci-k8s-vault.yml'

variables:
    # Secrets managed by Vault
    K8S_DEFAULT_KUBE_CONFIG: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/kubernetes/noprod?field=kube_config"
    K8S_PROD_KUBE_CONFIG: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/kubernetes/prod?field=kube_config"
    VAULT_BASE_URL: "https://vault.acme.host/v1"
    # $VAULT_ROLE_ID and $VAULT_SECRET_ID defined as a secret CI/CD variable

GitLab compatibility

ℹ️ This template is actually tested and validated on GitLab Community Edition instance version 13.12.11