Devops

Continuous Integration

Azure DevOps

anonlink-entity-service is automatically built and tested using Azure DevOps in the project Anonlink <https://dev.azure.com/data61/Anonlink>.

It consists of a build pipeline <https://dev.azure.com/data61/Anonlink/_build?definitionId=1>.

The build pipeline is defined in the script azure-pipelines.yml which uses resources from the folder .azurePipeline.

Azure CI Stages

Overview of CI stages showing dependencies and indicative run times.

The continuous integration stages are:

  • building and pushing the following docker images: - the frontend data61/anonlink-nginx - the Python base image data61/anonlink-base - the backend data61/anonlink-app - the tutorials data61/anonlink-docs-tutorials (used to tests the tutorial Python Notebooks) - the benchmark data61/anonlink-benchmark (used to run the benchmark)
  • runs the benchmark using docker-compose and publishes the results as an artifact in Azure
  • runs the tutorial tests using docker-compose and publishes the results in Azure
  • runs the end to end tests by deploying the whole service on Kubernetes, running the tests found in backend/entityservice/tests and publishing the results in Azure. The pod logs are also available in Azure DevOps.

The build pipeline is triggered for every push on every branch. It is not triggered by Pull Requests to avoid duplicate testing and building potentially untrusted external code.

The build pipeline requires two environment variables provided by Azure environment:

  • dockerHubId: username for the pipeline to push images to Data61’s Docker hub account.
  • dockerHubPassword: password for the corresponding username (this is a secret variable).

It also requires a service connection to a k8s cluster to be configured.

Base Image

The CI system builds and pushes the base image, before building downstream images. The CI system builds the application images using the current base VERSION. If a base image with the given digest is already present on Docker Hub the base image won’t be rebuilt.

For additional details see Dependencies.