Dynamically Retain Azure DevOps Pipelines

Update: Can now find this YAML on the TheYAMLPipelineOne repository

Introduction

Azure DevOps Pipelines can be very powerful, even more so when coupled with Multi-Stage Pipelines. If new to this check out my post on starting on Multi Stage YAML Pipelines. One of the things I like to do with my pipelines is combine CI/CD into the same pipeline. This coupled with multi-stages means that we have to be mindful of our pipeline retentions. This article will cover how to automatically retain Azure DevOps Pipelines. If you’d like to view the YAML pipelines with template check out my repo TheYAMLPipelineOne.

The Problem

What I am saying is our last stage is usually a production environment; however, not all pipelines will make it to the production stage. This could be due to testing issues, additional feedback, etc.. ADO has pipeline retention settings and we may not want to keep every pipeline execution. However, we should always keep copies of what was previously deployed to production for auditing and rollback purposes. So, the question becomes how can we leverage ADO to dynamically retain only Azure DevOps Pipelines that deployed to a production stage?

Requirements

Usually, we only need to retain pipelines that deployed to a production stage. This means we will need to have this process conditionally load if there is a production stage deployment was successful. I typically use the same pipeline for my CI/CD and load environment stages based on the Build.Reason or alternatively this could be done based on branch. For more on this check out my Omaha Azure User Group Presentation on YAML Deployment Pipelines.

For the purposes of this walkthrough let’s say we want to retain all production deployments for 2 years.

To only run for production deployments a condition like this could be involved:

    - ${{ if eq(environmentName,'prd') }} :

This condition will run a task which I got from a Microsoft Doc.

- task: PowerShell@2
  condition: and(succeeded(), not(canceled()))
  name: RetainOnSuccess
  displayName: Retain on Success
  inputs:
    failOnStderr: true
    targetType: 'inline'
    script: |
      $contentType = "application/json";
      $headers = @{ Authorization = 'Bearer $(System.AccessToken)' };
      $rawRequest = @{ daysValid = 365 * 2; definitionId = $(System.DefinitionId); ownerId = 'User:$(Build.RequestedForId)'; protectPipeline = $false; runId = $(Build.BuildId) };
      $request = ConvertTo-Json @($rawRequest);
      $uri = "$(System.CollectionUri)$(System.TeamProject)/_apis/build/retention/leases?api-version=6.0-preview.1";
      Invoke-RestMethod -uri $uri -method POST -Headers $headers -ContentType $contentType -Body $request;

Templating

If you have worked with me or follow me then you will realize the need to template this. I originally was going to load this into a job but honestly the production stage should fail or pass in its entirety and this process shouldn’t interrupt a production deployment in the unlikely event of a failure.

Additionally trying to embed this task in an existing deployment job or appending it as a job would be a struggle if trying to maintain parallel jobs and optimizing processing time. Thus, a stage that will run post production deployment seems to make sense.

ado_retain_pipeline_stage.yml

stages:
- stage: retain_pipeline
  jobs:
  - template: ../jobs/ado_retain_pipeline_job.yml

Nothing here really besides a wrapper for the job template.

ado_retain_pipeline_job.yml

jobs:
- job:
  steps:
  - template: ../tasks/ado_retain_pipeline_task.yml

Again, nothing here to worry about parameters since we will be using system variables. I suppose one could pass in days for retention; however, this type of property might be one that is better fit for a universal setting…..and by using templates we can define this in one spot.

ado_retain_pipeline_task.yml

steps:
- task: PowerShell@2
  condition: and(succeeded(), not(canceled()))
  name: RetainOnSuccess
  displayName: Retain on Success
  inputs:
    failOnStderr: true
    targetType: 'inline'
    script: |
      $contentType = "application/json";
      $headers = @{ Authorization = 'Bearer $(System.AccessToken)' };
      $rawRequest = @{ daysValid = 365 * 2; definitionId = $(System.DefinitionId); ownerId = 'User:$(Build.RequestedForId)'; protectPipeline = $false; runId = $(Build.BuildId) };
      $request = ConvertTo-Json @($rawRequest);
      $uri = "$(System.CollectionUri)$(System.TeamProject)/_apis/build/retention/leases?api-version=6.0-preview.1";
      Invoke-RestMethod -uri $uri -method POST -Headers $headers -ContentType $contentType -Body $request;

This is just copying the Microsoft source from earlier

End Result

In the above can see there is a build, dev, tst, and prd stages followed by the retain pipeline stage. This retention can then be confirmed by looking at the retention leases on the pipeline

Conclusion

There you go! This is one way to retain Azure DevOps Pipelines and is fairly simple to implement. Additionally, if you adhere to the template strategy, I laid out then it’s just adding that one line of code to your pipeline to bring in the additional retain_pipeline stage.