Have you tried creating a continuous integration environment in Azure DevOps (formerly VSTS)?

Isn’t it a crazy challenge?

I spent a couple of weeks trying to track down the information so that I can actually have a complete working pipeline.

As a result, it no longer has to be so hard for you…

I documented everything so that any automation engineer can use Azure DevOps to create a continuous integration environment for automation.

Why DevOps?

value of dev ops

value of dev ops

How to create your first CI build chain

  1. Navigate to your VSTS account and team project. Click Pipelines
  2. Select New pipeline to create a new build definition.

Configuring Github

  1. Choose GitHub from Where is your code
  2. Authorize communication with Github by clicking the Authorize button
  3. Select your repository

Executing the pipeline

  1. Select the ASP.NET Core (.NET Framework) build template
  2. A .yml file will be created for your repository
  3. Click Save and Run

Adding continuous integration

  1. Select Builds > Edit > Triggers. Under Continuous integration, select on the name of your repository.
    1. Toggle on the checkbox labeled Enable continuous integration.
    2. You can select CI for both merges and Pull Requests – https://prnt.sc/kb0g6n

How to create scheduled builds

  1. Picking up from the section above, click the Add button under Scheduled section
  2. Update the time and the day of when you want to run your builds
  3. Save & queue

How to set up your test .dll paths

Docs from MS on file matching patterns

The hardest time I had was how to configure my paths for my automation .DLL files. It’s not intuitive to know what is the working directory of your code base.

Here are some examples of what’s worked for me:

Executable Paths

Column A shows the local path. Column B shows how the path looks in Azure DevOps.

How to pass parameters to test code from a build or release pipeline?

A: Use a runsettings file to pass values as parameters to your test code. For example, in a release that contains several stages, you can pass the appropriate app URL to each the test tasks in each one. The runsettings file and matching parameters must be specified in the Visual Studio Test task.


How to add a status badge to your Github repo

The instructions on this are really good from Microsoft and you can follow them here in the Get the status badge section

All about YAML

Super important YAML schema

Working YAML file for UI Automation

Possibly a useful YAML for test automation

I found this YAML snippet for executing tests with VSTest from here

Not sure if it’s helpful yet, but it might be.

Posh script

# Create a secret variable
- powershell: |
Write-Host '##vso[task.setvariable variable=mySecret;issecret=true]abc'

# Attempt to output the value in various ways
- powershell: |
# Using an input-macro:
Write-Host "This works: $(mySecret)"

# Using the env var directly:
Write-Host "This does not work: $env:MYSECRET"

# Using the mapped env var:
Write-Host "This works: $env:MY_MAPPED_ENV_VAR"
MY_MAPPED_ENV_VAR: $(mySecret)

How to set and read environment variables?

I literally spent days of research, trial, and error trying to figure out how to set and read environment variables from an Azure test agent.

Regardless, I figured it out. Enjoy the solution.

  1. First you need to create some environment variables in your Azure DevOps UI that you want to use for values. This is an example of a variable that I would like to set on the test agent for my automation scripts. sauce.userName
  2. I will use the value of this variable(sauce.userName) and set it in my Environment Variables of the test agent when my automation is running. That way, the value of this variable isn’t exposed to the public. Well, it’s exposed to you because I am writing this post haha. But pretend it’s some “critical information” like your credit card 🙂
  3. Next, you will want to create a Powershell script that you attach to your solution. Here’s my solution layout.

    .ps1 solution structure

    .ps1 solution structure

  4. Here is what you want in your Powershell script. Basically, I’m going to pass in two variable values and then my Powershell script will set the Environment Variables on the agent for the user.

    1. Finally, you want to have a powershell step in your YAML that executes this Powershell script and passes in the values from the variables that you set in the Azure DevOps UI. This is what my YAML step looks like and I’m basically setting the SAUCE_USERNAME and SAUCE_ACCCESSKEY variables. In order for the YAML to understand where these variables come from, you need to convert sauce.userName to SAUCE_USERNAME. That’s the Azure convention. Read more about working with variables.

      powershell to set env variables

      powershell to set env variables

How to create your first project

  1. Go here
  2. Click the Get Started For Free button

How to run NUnit tests in VSTS

Filtering Tests

Instead of using Nunit test filters, you need to use the MSTest filters. For example, you need to convert:

cat == unit to /TestCaseFilter:”Category=unit”

Example of MStest filter

This will execute all Nunit tests that have the Category “unit”.

Azure DevOps resources

Videos from .NET Conf 2018


%d bloggers like this: