GitLab Pipelines

GitLab Pipelines#

How to create an automation script in GitLab - Or at least how to get started…

How to Handle Automation Scripts#

In GitLab, a Project has one central YAML file to configure automation, the .gitlab-ci.yml file. This file needs to be under version control and must reside in the root directory of the repository

Although GitLab considers only the .gitlab-ci.yml file for defining the “main” Pipeline, you can still set up multiple pipelines. This can be achieved by establishing conditions for the individual steps within the main Pipeline or by including “child” Pipelines.

Minimal Content#

A GitLab Pipeline must include at least the following elements:

stages

is mandatory top-level key containing a list of strings that identify the sequence of stages the Pipeline will sequentially run through.

Stages are processed sequentially and define the order of job execution in a GitLab Pipeline, helping to organize the workflow from start to finish.

Each stage groups jobs that execute concurrently. Stages allow you to manage the flow of your Pipeline, ensuring each step completes before moving on to the next.

You can choose the name of a stage freely, but some typical names are:

  • “build”: Compile code, build binaries, or prepare artifacts.

  • “test”: Run automated tests to ensure code quality.

  • “deploy”: Deploy your application to staging or production.

Jobs in the next stage will only start once the previous stage has successfully been completed. For more details on managing stages, refer to GitLab’s stages documentation.

<job_name>

are top-level keys with arbitrary names. Jobs are the individual elements that make up a GitLab CI/CD pipeline.

Jobs in GitLab run based on their assigned stage and can execute independently or sequentially within the pipeline.

Some essential properties for configuring jobs:

  • script: Specifies the shell commands to run.

  • stage: Defines the stage the job belongs to.

  • only/except: Defines conditions for job execution (e.g., run only on specific branches).

  • artifacts: Defines files to pass between jobs or to retain after a job completes.

  • needs: Specifies dependencies with other jobs in the pipeline.

  • rules: Advanced logic for conditional job execution.

Jobs in GitLab are flexible and allow extensive customization. Check GitLab’s documentation on jobs for more information.

Note

In the future the run key might become an alternative to the script key.

script

is a mandatory key within a job.

script defines a list of shell commands that will be run sequentially when the job is executed.

Each command in the script can be a single line or a multi-line block, allowing for complex operations to be performed during the job.

A Minimal Example#

An minimal exemplary Pipeline file could look like this:

# .gitlab-ci.yml

stages:
  - greet

greet_job:
  stage: greet
  script:
    - echo "Hello, World!"

How to Trigger Pipelines#

In GitLab, Pipelines can be triggered by various events, and you have the option to set rules that determine whether an event should initiate any action, both at the Pipeline level and for each individual job.

To define rules for the entire Pipeline, they must be placed under the rules key within the top-level key workflow.

For instance, a rule to trigger a Pipeline on a Merge Request could look like this:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

For individual jobs, the rules keyword can specify whether an event should trigger a particular job.

For example, a rule to trigger a job on a Merge Request could look like this:

job1:
  script:
    - echo "This is a merge request"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

How to Define Variables#

GitLab allows the use of variables in the definition of Pipelines and provides a variety of predefined variables.

We can differentiate between:

  • Predefined Variables: GitLab provides a set of predefined variables that are automatically available in your Pipeline jobs. These variables typically start with CI_... and contain information about the Project, Pipeline, job, and more.

Warning

Sensitive data should be stored outside of GitLab! Half-way acceptable is to use protected and masked custom variables

  • Custom Variables: You can define custom variables at the Project, Group, or Subgroup level. These variables can be set via the Web-UI and can be used to store configuration values or secrets that your jobs need to access.

An exemplary use of a predefined variable:

myjob:
  stage: test
  script:
    - echo "This job run in stage: '$CI_JOB_STAGE'"

Refer to the official documentation for a complete overview of GitLab’s predefined variables and their availability.

Some Advanced Features#

Pipeline Triggers:

While in GitLab single jobs can be triggered via rules it is also possible to set rules for the entire Pipeline using the top-level keyword workflow.

workflow:
  rules:
    - if: $CI_COMMIT_TITLE =~ /-draft$/
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

In fact, with workflow you can control more than just when the Pipeline should be, for example also when it should be canceled.

Read more about controlling the Pipeline behavior with workflow.

Using Variables:

GitLab provides predefined variables that provide information on things like:

  • CI_PIPELINE_SOURCE: Indicates how the Pipeline was triggered.

  • CI_PROJECT_ID: The ID of the related project.

  • …

GitLab allows you to define and use variables in Pipelines, both on the top level and within a job:

variables:
  GLOBAL_VAR: "I'm global"

job1:
  variables:
    SOME_VAR: "I'm local"
  script:
    - echo "Variables - '$GLOBAL_VAR' and - '$SOME_VAR'"

job2:
  script:
    - echo "Variables '$GLOBAL_VAR' and '$SOME_VAR'"  # SOME_VAR will be ''

In addition, you can set variables in the UI, which is the recommended approach to store sensitive information, like access tokens or password.

Using before_script and after_script:

You can define commands that run before or after the main script commands even if the script command fails or times out.

job_name:
  before_script:
    - echo "This runs before the main script"
  script:
    - echo "This is the main script"
  after_script:
    - echo "This runs after the main script"

Read more about before_script and after_script.

Setting Defaults:

The top-level key default can be used to set values and behaviors that will apply to all jobs, unless overwritten directly.

Some Candidates for defaults are:

  • after_script: Commands to run after script

  • before_script: Commands to run before script

  • tags: Set of tags to select a runner

  • image: The Docker image to use

  • timeout: Maximal runtime of a job

Creating Dependent Jobs:

You can define job dependencies using the needs keyword, ensuring that jobs run in sequence even within the same stage.

test1:
  stage: test
  script:
    - echo "Running test 1"
test2:
  stage: test
  needs: ["test1"]
  script:
    - echo "Running test 2"
Parallel Execution:

You can run a job multiple times optionally with different configurations (e.g., testing across different versions of a programming language).

test:
  parallel:
    matrix:
      - python_version: [3.7, 3.8, 3.9]
  script:
    - echo "Testing with Python version $python_version"

Read more about the parallel keyword.

Caching Dependencies:

Caching is a key feature to create fast pipelines and reduce computational overhead.

The job keyword cache can be used to determine what should be cached in a job.

build:
  script:
    - echo "Hello" >> hello.txt
  cache:
    paths:
      - hello.txt

Read more about Caching.

Using Services:

If your job requires a service like a database, you can define service containers that will run alongside the job.

test:
  services:
    - name: postgres:12
      alias: db
      entrypoint: ["/usr/local/bin/db-postgres"]
      command:
        - start
  script:
    - echo "Tests with PostgreSQL"

Read more about services in the GitLab documentation.

Using Tags (Targeting Runners):

You can specify which runners to use for a job based on the runner’s tags.

job_name:
  tags:
    - docker
  script:
    - echo "Running on a runner with the 'docker' tag"

Note

These tags are not in any way related to tags.

Reusing Configurations:

Use YAML anchors to reuse configurations across jobs.

.default_script: &default_script
  - echo "This is a default job template"

job1:
  script:
    - *default_script
    - echo "This is job 1, extending the default script


.job_template: &job_default  # .job_template is hidden
  image: ruby:2.6
  services:
    - postgres

job2:
  <<: *job_default
  script:
    - echo "This is job 2, extending the default job template"
Content