Pipelines Templatization

Introduction

Templatization of pipelines is a powerful feature that enables the reuse of pipeline configurations across multiple pipelines and even projects, ensuring consistency, security and efficiency in the CI/CD process. This approach is supported by various CI/CD tools especially Azure DevOps, allowing teams to define standard pipeline step configurations that can be used across all pipelines in an organization.

The DRY (Don’t Repeat Yourself) principle

The DRY (Don’t Repeat Yourself) principle is a fundamental concept in software development that emphasizes the importance of avoiding redundancy by creating reusable and modular components. This principle is highly applicable to CI/CD pipelines.

In the context of pipelines, the DRY principle advocates for the creation of templates that encapsulate common configurations and workflows. By adhering to this principle, development teams can eliminate redundancy, reduce errors, and enhance the maintainability of their CI/CD processes.

Parameters and Variables in Templates

The essence of the DRY principle in pipeline templatization lies in the creation of reusable configurations using parameters. These parameters serve as dynamic inputs that allow customization while maintaining a standardized structure.

Parameters

Parameters are the values that you can pass into a template to customize its behavior. They can be used to control the execution of steps, jobs, or stages within a template. Parameters can be of various types, including strings, booleans, and objects, and they can have default values. Let’s see it with an example :

parameters:
- name: nodeVersion
 default: '18.X'

steps:
- task: UseNode@1
displayName: Use Node version ${{ parameters.nodeVersion }}
inputs:
    version: ${{ parameters.nodeVersion }}

This template accepts the nodeVErsion as a parameter and uses ‘18.X’ as the default when the parameter is not provided.

Variables

Variables in pipelines especially in Azure DevOps are a powerful feature that allows to dynamically assign values to parts of the pipeline configuration. They can be defined at various scopes pipeline, stage and job levels, making them highly flexible.

User-Defined Variables

variables:
  myVariable: 'Hello World'

steps:
  - script: echo $(myVariable)

System Variables

Azure DevOps provides a set of predefined system variables that are automatically available in the pipeline. These variables provide information about the pipeline, agent, and environment.

steps:
  - script: echo $(Build.BuildId)

This step prints the ID of the current build, using the Build.BuildIdsystem variable.

Types of templates

As it is known, the common YAML pipeline in Azure DevOps is composed of stages, jobs and steps as shown in this example:

trigger:
- main

stages:
- stage: Build
 displayName: 'Build Stage'
 pool:
    vmImage: ubuntu-latest
 jobs:
 - job: BuildJob
    displayName: 'Build Job'
    steps:
    - script: echo Building the application...
      displayName: 'Build Application'
    - script: echo Running tests...
      displayName: 'Run Tests'

So the same for types of templates:

• Stage templates: Stage templates allow the definition of one or more stages that can be reused across multiple pipelines. For example, you might have a common stage for building, testing and deploying some applications.

parameters:
  - name: buildJobName

stages:
- stage: Build
 displayName: 'Build Stage'
 jobs:
 - job: Build
    displayName: 'Build Job'
    steps:
    - script: echo Building...

- stage: Test
 displayName: 'Test Stage'
 jobs:
 - job: Test
    displayName: 'Test Job'
    steps:
    - script: echo Testing...

- stage: Deploy
 displayName: 'Deploy Stage'
 jobs:
 - job: Deploy
    displayName: 'Deploy Job'
    steps:
    - script: echo Deploying...

To use this stage template in a pipeline with buildJobName as a parameter:

trigger:
  - main

pool: 'ubuntu-latest'

stages:
  - template: templates/build-test-deploy.yml
    parameters:
      buildJobName: 'Build Job'

• Job Templates: Job templates allow defining one or more jobs that can be reused across different stages within pipelines. Example of job template:

jobs:
- job: Test
 displayName: 'Test Job'
 steps:
 - script: echo Testing...

To use this job template in a stage:

trigger:
- main

pool: 'ubuntu-latest'

stages:
- stage: Test
 displayName: 'Test Stage'
 jobs:
 - template: templates/test.yml

• Step Templates: Step templates allow the definition of one or more steps that can be reused across different jobs within multiple pipelines. This is useful for defining common tasks, such as running linting checks and installing Node which are frequently used.

parameters:
  - name: nodeVersion

steps:
  - task: UseNode@1
    displayName: Use Node version ${{ parameters.nodeVersion }}
    inputs:
      version: ${{ parameters.nodeVersion }}

  - task: Cache@2
    inputs:
      key: 'npm | "$(Agent.OS)" | package-lock.json'
      restoreKeys: |
        npm | "$(Agent.OS)"
      path: $(npm_config_cache)
    displayName: Cache npm

  - task: Bash@3
    displayName: Install NodeJS Dependencies
    inputs:
      targetType: inline
      script: |
        npm ci
      workingDirectory: $(System.DefaultWorkingDirectory)

To use this steps template in the main pipeline:

trigger:
- main

pool: 'ubuntu-latest'

stages:
- stage: BuildAndTest
 displayName: 'Build and Test Stage'
 jobs:
 - job: NpmInstall
    displayName: 'Npm Install'
    steps:
    - template: templates/npm-install-step.yml
      parameters:
        nodeVersion: '20.X'

• Variables Template: The variables template can be defined by specifying the variables and their values in a separate YAML file. For example:

variables:
  - name: nodeVersion
    value: '18.X'

and it can be used like this:

variables:
    - template: 'templates/variables.yml'

trigger:
- main

pool: 'ubuntu-latest'

stages:
- stage: BuildAndTest
 displayName: 'Build and Test Stage'
 jobs:
 - job: NpmInstall
    displayName: 'Npm Install'
    steps:
    - template: templates/npm-install-step.yml
      parameters:
        nodeVersion: ${{ variables.nodeVersion }}

Benefits of Pipeline Templatization

• Consistency and Standardization: Templatization ensures that all pipelines within an organization follow the same structure and conventions. This consistency is crucial for maintaining quality and reducing the risk of errors.
• Reduced Duplication: By using templates, the pipelines are cleaner and easier to manage but also reduce the likelihood of errors.
• Enhanced Maintainability: When changes need to be made, such as updating a step or adding a new parameter, you only need to update the template.
• Flexibility and Customization: Templates can be customized with parameters, allowing the adaptation of the pipeline’s behavior to the specific needs of each project or stage.

In summary, the templatization of pipelines in the context of CI/CD processes, offers a multitude of benefits that enhance efficiency, sustainability, and effectiveness.