GitHub Workflows

GitHub Workflows#

How to create an automation script in GitHub? - Or at least how to get started…

How to Handle Automation Scripts#

Automation scripts for GitHub should be included in the repository and must be located in the .github/workflows/ directory. You can name the YAML files as you prefer, and you are free to add multiple files to the .github/workflows/ directory, enabling you to associate several Workflows with a single Project. Each file in that directory is treated as a separate definition of a Workflow. This is an excellent method for sharing complete workflows across projects; simply copy the relevant YAML file.

Minimal Content#

A GitHub Workflow must include at least the following elements (keys):

Note

GitHub provides a comprehensive overview of all top-level keys and their purpose.

on

is a mandatory top-level key that determines when to initiate the Workflow, such as when pushing code, opening a pull request, or creating an issue.

Workflows in GitHub can be triggered by:

  • Repository events: Like commits, pull requests, or issue creation, e.g. on: create or on: [push, fork].

  • External events: Webhooks or APIs that signal GitHub to start a Workflow.

  • Scheduled times: Set Workflows to run at regular intervals (e.g., nightly builds using cron syntax).

  • Manual triggers: Start a Workflow manually from the GitHub UI.

We recommend to read about triggering events if you want to learn more about them.

jobs

is a top-level key that holds a dictionary of jobs, each specify an environment along with a sequence of tasks to perform.

Jobs can be arbitrarily named (i.e. the key can be set freely). They will run in parallel by default and each job can contain a collection of steps that are executed sequentially.

Some of the more relevant keys a job might contain:

  • runs-on: Defines the type of machine (or VM) to run this job on.

  • steps: Sequence of tasks to perform.

  • needs: Establishes dependencies with other jobs.

  • if: Allows to specify conditions to prevent a job from running.

  • permissions: Modify the default permissions granted to the GITHUB_TOKEN

jobs might look like this:

jobs:
  my-job:
    name: My Job
    runs-on: ubuntu-latest
    steps:
      - name: Print a greeting
        uses: actions/checkout@v3
      - name: Print a greeting
        run: |
          echo ${{ github.repository }}

A full list of keys a job accepts can be found here.

steps

is a mandatory key within a job. steps defines a list of individual commands or actions to be executed as part of a job.

Steps can either run shell commands or use pre-built GitHub Actions to perform specific tasks.

Some keys an entry in steps might contain:

  • if: Allows to specify conditions to prevent a step from running.

  • run: A string that will be run as a shell command.

  • uses: Specifies a pre-defined action to run.

A Minimal Example#

An minimal exemplary Workflow file could look like this:

# .github/workflows/hello_world.yml

on:
  push:
    branches:
      - main

jobs:
  say_hello:
    steps:
      - uses: actions/checkout@v2
      - run: echo "Hello, World!"

How to Trigger Workflows#

In GitHub, a variety of events can trigger a Workflow, and each Workflow (i.e., each YAML file located in .github/workflows/) must include the top-level key on, which specifies the events that will activate this Workflow.

Additionally, individual jobs (i.e., entries under the top-level key jobs) and specific steps (i.e., elements within the steps key of a job) can utilize the if key to set conditions for skipping that particular step or job. However, it’s important to note that if a Workflow runs and a job is skipped, its status will be updated to skipped.

A conditional job could look like this:

name: example-workflow
on: [push]
jobs:
  conditional-greet:
    if: ${{ github.repository == 'myuser/myrepo' }}
    runs-on: ubuntu-latest
    steps:
      - run: echo "Howdie!"

How to Define Variables#

GitHub allows the use of variables in Workflows definitions and provides a variety of context variables that can be accessed using the ${{...}} syntax.

Some important contextual variables include:

  • github: Provides metadata about the Workflow run, Repository, and event that triggered the Workflow, etc.

    • E.g. github.event_name indicates the name of the event that triggered the Workflow (e.g. push, pull_request, etc.) while github.repository provides the name of the Repository.

  • vars: Contains variables defined at the Repository, Organization or Environment level. These variables can be set via the Web-UI.

    • For instance, vars.my_variable represents a variable named my_variable.

Warning

Sensitive data should be stored outside of GitHub! secrets is the only half-way acceptable place to store sensitive data!

  • secrets: contains the names and values of secrets defined for the Repository, Organization or Environment. Secrets are handled in a specific way by GitHub and can be set via the Web-UI.

    • For example, secrets.TOKEN refers to a secret named TOKEN.

Refer to the official documentation for a complete overview of GitHub’s contextual variables and their availability.

Inspect context variables

Context variables are, as their name suggests, context-dependent, and it may not always be clear what values are actually defined when a Workflow runs.

A straightforward way to debug context variables is to simply have their content returned by a job that runs in specific context:

jobs:
  list-github-context:
    runs-on: ubuntu-latest
    steps:
      - name: Print GitHub Context Variables
        env:
          GITHUB_CONTEXT: ${{ toJSON(github) }}
        run: |
          echo "GitHub context variables:"
          echo "$GITHUB_CONTEXT" | jq '.'

Some Advanced Features#

${{...}} Context Variables:

As the name suggests, context variables hold contextual information and vary significantly under different running conditions.

Some important contextual variables are:

  • github: Provides metadata about the Workflow run, repository, and event that triggered the workflow.

  • vars: Contains variables defined on Repository, Organization or Environment level.

  • secrets: Names and values of available secrets.

Refer to the official documentation for a complete overview of GitHub’s contextual variables and their availabilities.

Inspect context variables

jobs:
  list-github-context:
    runs-on: ubuntu-latest
    steps:
      - name: Print GitHub Context Variables
        env:
          GITHUB_CONTEXT: ${{ toJSON(github) }}
        run: |
          echo "GitHub context variables:"
          echo "$GITHUB_CONTEXT" | jq '.'
On-the-Fly Variables

Variables can be generated as part of a Workflow (e.g., step or job outputs) can be referenced by other steps.

jobs:
  job1:
    runs-on: ubuntu-latest
    outputs:
      output1: ${{ steps.mystep.outputs.test }}
    steps:
      - id: mystep
        run: echo "test=hello" >> "$GITHUB_OUTPUT"
  job2:
    runs-on: ubuntu-latest
    needs: job1
    steps:
      - env:
          OUTPUT1: ${{needs.job1.outputs.output1}}
        run: echo "$OUTPUT1"
Expression Evaluation:

GitHub allows evaluating expressions in the ${{...}} syntax.

jobs:
  example:
    runs-on: ubuntu-latest
    steps:
      - name: Set a message based on event type
        run: |
          MESSAGE="${{ github.event_name == 'push' && 'This is a push event!' || 'This is not a push event.' }}"
          echo $MESSAGE
Using Workflow Templates:

  • GitHub offers pre-built workflow templates that can be used as-is or customized to meet your project needs.

  • These templates simplify common tasks such as:

    • Testing code on different platforms or environments.

    • Deploying applications to various hosting services.

    • Scanning for security vulnerabilities or code quality issues.

Storing Secrets:

Use GitHub Secrets to securely store sensitive information, like API keys or credentials. Secrets are encrypted and only available at runtime.

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: ssh -i ${{ secrets.SSH_PRIVATE_KEY }} user@server.com 'bash deploy.sh'
Creating Dependent Jobs:

You can define job dependencies using the needs keyword, ensuring that jobs run in sequence.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - run: echo "Building project"

  test:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - run: echo "Running tests"

  deploy:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - run: echo "Deploying to production"
Using a Matrix:

You can run a job multiple times with different configurations (e.g., testing across different versions of a programming language).

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.7, 3.8, 3.9]
    steps:
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - run: python --version
Caching Dependencies:

Speed up workflows by caching dependencies or files that are used across runs.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Cache node modules
        uses: actions/cache@v3
        with:
          path: node_modules
          key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}
Using Databases and Service Containers:

If your workflow requires a service like a database, you can define service containers that will run alongside the job.

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:12
        ports:
          - 5432:5432
        options: >-
          --health-cmd "pg_isready"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
      - run: psql --version
Using Labels (Targeting Runners):

You can specify which runners to use for a job based on the runner’s labels (e.g., self-hosted, ubuntu-latest).

jobs:
  build:
    runs-on: self-hosted
    steps:
      - run: echo "Building project on a self-hosted runner"
Reusing Workflows:

Use reusable workflows to call another workflow from within a workflow, reducing duplication of common tasks.

jobs:
  call-another-workflow:
    uses: ./.github/workflows/reusable-workflow.yml
    with:
      param1: value1
Using Environments:

You can define environments with specific secrets and protection rules for different stages of deployment (e.g., development, staging, production).

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://my-app.com
    steps:
      - run: echo "Deploying to production"
Content