🤗 Embrace automation 🤗

🤗 Embrace automation 🤗#

Recap#

Getting started with CI/CD is easier than it sounds—really, all you need is ❤️… - well, a YAML file 😉. Begin with small, simple automation scripts to familiarize yourself with CI/CD fundamentals, such as automating tests or compiling code. Setting clear trigger points for these scripts, such as on code pushes or pull requests, will help manage when your workflows run.

For projects involving non-sensitive data, remote hosted runners are convenient and require minimal setup, while self-hosted runners offer more control and can be eady to configure.

To enhance reproducibility and flexibility, consider using Docker containers in your automation scripts. They provide a consistent and isolated environment for your jobs, reducing the risk of environment-related issues. As you gain confidence, feel free to iterate on your automation scripts to improve efficiency and functionality.

However, always prioritize security. Runners have access to your code and environment, making it essential to secure them properly. Store sensitive data only in custom variables or the secrets context via the Web UI to ensure your project remains protected.

Playground Repository

Use a private repository to experiment with automation scripts and workflows.

Tip

Have a (private) template repository with a basic CI/CD setup ready to go for new projects.

Feature Branch Approach - with Automation#

The Feature Branch Approach has been a recurring topic, first discussed in the working-with-git course, where we focus on the -side of things. We also explored this approach in git-and-its-remotes, examining how it can be implemented using remote service features, such as Issues, and Merge/Pull Requests. Here, we will focus on integrating automation into this approach to streamline workflows and adhere to best practices.

In short, the Feature Branch Approach is a development strategy where individual features or enhancements are developed in isolated branches. This allows teams to work concurrently without affecting the healthy reference (i.e. the main branch in most cases) until the features are ready for integration.

Throughout this process, various events may trigger a CI/CD workflow. We recommend identifying the events that are the most useful to your specific project and setup.

To help you get started, here are some suggestions:

Test the development branch#

You can setup tests to be triggered on all branches except the main branch or for any push event to a branch with an open Merge/Pull Request. By doing this, you can ensure that issues in the development branch are spotted early on.

Report code coverage in the Merge/Pull Request#

Both GitHub and GitLab make it relatively easy report test coverage. You can configure your automation script performing the tests to also report changes in test coverage directly within the Merge/Pull Request, providing valuable feedback to other collaborators.

Perform linting checks on the development branch#

Implement linting checks on the development branch (e.g., of a new feature) to enforce coding standards and catch potential issues before they merged into the main branch. This proactive approach can you can significantly improve code quality and maintenance.

Test an anticipated merge commit#

When a Merge/Pull Request is no longer a draft, you can run an automation script that performs a merge (without committing it) of the feature branch, tests an reports the results.

This ensures that the merged code passes all tests before final integration. Allow merging only if this test succeeds.

In GitLab such a test could look like this:

# .gitlab-ci.yml
stages:
  - merge

variables:
  GIT_STRATEGY: fetch

merge_and_test:
  stage: merge
  script:
    - echo "Anticipating merge commit and testing..."
    - git fetch origin main
    - git checkout -b temp-merge-branch
    - git merge --no-commit origin/main
    - echo "Running tests after merge..."
    - # run your tests here
  rules:
    - if: '$CI_MERGE_REQUEST_ID && $CI_MERGE_REQUEST_DRAFT == "false"'
  when: manual  # Remove it the job should run automatically

Perform a version increase upon merge#

You can setup a process that automatically creates a new tag as soon as a feature branch is merged. This process might fetch the latest tag, increment its minor version number by 1, and push a new tag. Alternatively, you can use existing tools, such as semantic-release for GitHub to perform this task automatically for you.

Cleanup branches after merge#

Automating the cleanup of branches that have been merged and are no longer needed is a good practice.

Tip

This is also a relatively easy automation task to start with!

Automate deployment to production or staging environment#

If you are working on a project that can be deployed, like a static website in GitHub- or GitLab- pages, you can setup a deployment script that runs on every update (i.e., push) to your healthy reference.

Note

The very content you are watching right now is depolyed in this manner! Checkout the pages.yml Workflow to see how exactly this is done. ``

Some considerations#

Start Small!#

CI/CD functionalities in GitLab and GitHub are excellent tools for speeding up, standardizing, and streamlining a variety of tasks. However, adding complexity through pipelines or workflows can quickly increase the intricacy of your scripts. Before you know it, you may find yourself navigating a maze of dependencies, conditions, and edge cases that can consume a significant amount of time to resolve and manage.

Therefore, before implementing any automation procedures, evaluate your workflow design and ensure it is as simple and straightforward as possible. The fewer dependencies, the better!

Another important consideration is to avoid having multiple “sources of truth.” This concept may seem unusual at first, but adhering to the principle that any parameterization (such as the version of a dependency) should be defined in only one location—especially when used across multiple stages in a workflow or pipeline—can greatly simplify management.

Finally, even when starting small, it’s essential to evaluate the structure of your project or repository before embarking on your automation journey. The various features and configurations offered by remote services are designed to work well with standard project structures for languages like Python, R, Scala, etc. Following typical project setups is likely to facilitate the use of automation functionalities, while maintaining custom folder structures and dependency patterns may lead to complications.

Create your own Docker images#

Both GitLab and GitHub leverage Docker extensively in their automation processes, and they make it easy for you to do create and distribute your own Docker images.

By creating custom Docker images, you not only establish isolated and reproducible environments for your projects but also drastically reduce the runtime of your automation scripts. This is because all installation and configuration of the environment can be completed during the image creation process.

Tip

For a comprehensive example of this approach, check out furrer-lab/r-containers.

Read more about creating Docker images on and

Debugging CI/CD Pipelines/Workflows#

When you start creating your own automation scripts, you may find that debugging them can be more challenging than debugging local code. This is partly because your YAML scripts are triggered by specific events that might not happen on demand. Additionally, both GitLab and GitHub offer a wide range of context-dependent variables, which can further complicate the debugging process.

A good approach for debugging is:

  • Debug the triggering process and the actual actions the script performs separately.

  • Choose an easy trigger to debug your Pipeline/Workflow (e.g., on push to a development branch).

  • Use echo statements to inspect variables.

Inspect context variables

jobs:
  list-github-context:
    runs-on: ubuntu-latest
    steps:
      - name: Print GitHub Context Variables
        env:
          GITHUB_CONTEXT: ${{ toJSON(github) }}
        run: |
          echo "GitHub context variables:"
          echo "$GITHUB_CONTEXT" | jq '.'