Advanced DevOps: PaaS service integration testing

In my last article we investigated integration tests with various underlying platform services like databases or message brokers. Time to make the next step in our journey towards Advanced DevOps.

In the Advanced DevOps series, I cover topics from continuous integration to continuous delivery, and monitoring. Let's go beyond build and test!

By testing against a matrix of supported platforms we are at a good start and it might even be sufficient for some applications or open-source projects. However, we very rarely run nowadays on top of on-premises or self-hosted databases but PaaS services from a cloud provider instead.

Many databases that we know and love in the on-prem world exist in the cloud as well. However, they are never identical as some features cannot be supported in a cloud context, e.g., for security reasons. Other cloud data services might not even have an on-premises variant. It can be assumed that this will be the case in the future even more often.

So, the next step is to run integration tests in our pipeline that validates the integration with underlying cloud services that our code depends on.

This is still not a full end-2-end system test where we deploy our full application into the cloud and run black-box tests against it. This is a white-box test executed as part of a regular build pipeline.

Our tool of choice is again Azure Pipelines and I have an example Java application that leverages Spring Data Gremlin to access Azure Cosmos DB. The application has a small set of Junit integration tests for Cosmos DB defined.

Create the test infrastructure

There are several strategies how to handle the infrastructure for such integration tests. The simplest one is to have a permanent static service instance that is used by the test runs. However, it may be complicated or even impossible to develop the tests in a way that parallel runs won't interfere with each other.

The other extreme is to provision a full service instance per integration test run. That ensures full encapsulation but might generate higher cost and take quite bit of time as the cloud vendor needs to provision lots of infrastructure.

The middle way that I prefer is to use the internal virtual resource separation that most of these services have. Azure Cosmos DB has the concept of databases and containers inside the same account, Azure SQL Database has the concept of databases and schemas etc.

In this Cosmos DB example, I decided to create a fresh database for every test run.

The next decision is how to manage the test infrastructure. This depends a bit on the complexity of the underlying infrastructure. If multiple services are needed, it makes sense to use an infrastructure as code approach like Azure Resource Manager templates or terraform. In my simple example where Cosmos DB is the only service a script is more than sufficient.

We use the AzureCLI task for this purpose that already takes care of the Azure login for us compared to a vanilla script task that does not.

 1- task: AzureCLI@2
 2    displayName: "Setup CosmosDB gremlin database for integration test"
 3    inputs:
 4      azureSubscription: 'AzureKaizimmerm'
 5      scriptType: bash
 6      scriptLocation: inlineScript
 7      inlineScript: |
 8        az cosmosdb gremlin database create --account-name $(intTestCosmosAccount) \
 9          --name $(intTestCosmosDbName) --resource-group $(intTestResourceGroup)
10        az cosmosdb gremlin graph create --account-name $(intTestCosmosAccount) \
11          --database-name $(intTestCosmosDbName) --name $(intTestCosmosCollectionName) \
12          --partition-key-path "/address" --resource-group $(intTestResourceGroup)        

The task consumes a service connection (AzureKaizimmerm), which is similar to the SonarCloud connection we created previously. But this time it is a connection of the type Azure Resource Manager.

The script itself uses an array of variables that we have created previously. In this case a static definition, a variable group is a viable alternative as well:

 1variables:
 2  - name: mavenCacheFolder
 3    value: $(Pipeline.Workspace)/.m2/repository
 4  - name: mavenCacheOpts
 5    value: '-Dmaven.repo.local=$(mavenCacheFolder)'
 6  - name: intTestResourceGroup
 7    value:  'kaizimmerm'
 8  - name: intTestCosmosAccount
 9    value:  'kaizimmerm-sample'
10  - name: intTestCosmosDbName
11    value:  'test$(Build.BuildId)'
12  - name: intTestCosmosCollectionName
13    value:  'sample-graph'
14  - name: intTestCosmosUsername
15    value: '/dbs/$(intTestCosmosDbName)/colls/$(intTestCosmosCollectionName)'
16  - name: intTestCosmosEndpoint
17    value:  $(intTestCosmosAccount).gremlin.cosmos.azure.com

Extract credentials and run the integration tests

Our Java application has a maven profile with the maven-failsafe-plugin defined to execute the integration tests. This comes in handy to not interfere a normal build as the integration tests will only work if a service instance is available and the credentials are provided.

The first task shown below extracts the Cosmos DB access key and write it into a job scoped variable. This is done by the ##vso[task.setvariable variable=intTestCosmosKey;issecret=true]$keyCosmos snippet that is an Azure Pipelines macro and not valid shell syntax. That is why we have the echo in front to have a valid syntax. The value of the variable will be saved as a secret and by doing so is masked from the log.

After that we have the regular Maven task that we used in previous articles.

 1- task: AzureCLI@2
 2    displayName: "Retrieve CosmosDB key for integration test"
 3    inputs:
 4      azureSubscription: 'AzureKaizimmerm'
 5      scriptType: bash
 6      scriptLocation: inlineScript
 7      inlineScript: |
 8        keyCosmos=`az cosmosdb keys list --name $(intTestCosmosAccount) \
 9                --resource-group $(intTestResourceGroup) --query primaryMasterKey -o tsv`
10        echo "##vso[task.setvariable variable=intTestCosmosKey;issecret=true]$keyCosmos"        
11  - task: Maven@3
12    displayName: 'Maven verify and integration-test'
13    inputs:
14      mavenPomFile: "pom.xml"
15      mavenOptions: "-Xmx3072m"
16      javaHomeOption: "JDKVersion"
17      jdkVersionOption: "1.11"
18      jdkArchitectureOption: "x64"
19      publishJUnitResults: true
20      sonarQubeRunAnalysis: true
21      sqMavenPluginVersionChoice: 'latest'
22      testResultsFiles: "**/surefire-reports/TEST-*.xml"
23      goals: "verify -Pintegration-test -Dgremlin.endpoint=$(intTestCosmosEndpoint) -Dgremlin.username=$(intTestCosmosUsername) -Dgremlin.password=$(intTestCosmosKey) $(mavenCacheOpts)"

Cleanup test resources

The final task is to clean up our test resources after the run. The condition always is key here as the resources need to be cleaned up even if the test itself failed or has been canceled which is not the default behavior, i.e. if a task fails the pipeline normally skips the following tasks.

 1- task: AzureCLI@2
 2    displayName: "Delete CosmosDB gremlin database from integration test"
 3    condition: and(always(), eq('${{ parameters.cleanUpTestCosmos }}', true))
 4    inputs:
 5      azureSubscription: 'AzureKaizimmerm'
 6      scriptType: bash
 7      scriptLocation: inlineScript
 8      inlineScript: |
 9        az cosmosdb gremlin database delete --account-name $(intTestCosmosAccount) \
10          --name $(intTestCosmosDbName) --resource-group $(intTestResourceGroup) --yes        

There might be scenarios where we want to keep the test resources to investigate a failed build. For this purpose, it is possible to override a parameter for an individual build:

1parameters:
2- name: cleanUpTestCosmos
3  default: true
4  type: boolean

Here we go, integration tests against the PaaS service Azure Cosmos DB!

Next steps

As mentioned in the introduction this article is part of a series where we are going to dive deeper into the magic of DevOps technologies. So, the best next step I recommend is to wait for my next article or setup your own integration tests.

As always, the code of this example is over at GitHub.