microsoft / azure-pipelines-ephemeral-agents

Use Ephemeral agents to deploy to private azure resources in a virtual network

Ephemeral Pipelines Agents

When you want to deploy to Azure Resources that aren't exposed on the internet, and only accessible via a private network you are excluded from using Microsoft-hosted agents, and you need to maintain your pool of self-hosted agents.

Ephemeral pipelines agent eliminates the need to maintain self-hosted agents (for deployment purposes) and still be capable of deploying to private azure resources. Ephemeral pipeline agents run in an Azure Container Instance (no public IP address and access to the private network) which are created on a needed basis to run a single a pipeline job.

This repo has several components that allow deployments to private virtual networks using short-lived agents and without having to maintain a permanent pool of self-hosted agents.

The purpose of this task is to create a short-lived Azure Pipelines Agent to run a deploy in a private virtual network so you can deploy assets to Azure Resources that are not internet accessible.

This way, you can deploy to private Azure resources without having to expose them on the internet or having to maintain self-hosted agents on the same (or with access) virtual network.

How it works

This technique relies on three requirements

1. A docker image that can run an Azure Pipelines agent in a container
2. Provision, an ephemeral agent, to run a deployment job (once)
   • While this can be done manually with a task, we have created a task that provisions, configures and registers an agent on a Azure Container Instance. The task works in symbiosis with the docker image.
3. The container runs a single pipeline job (and only one), unregisters the agent and deletes the container.
   • While an external component can do this, we have delegated the operation to the container. It can self destruct to eliminate moving parts and external components.

Your pipelines have to be structured the following way. It has (at least) two jobs:

• The first job uses this task and provisions and configures an ephemeral agent in a specific pool of your choice.
• The second job (runs after the first one has finished) does the actual deployment. It runs on the pool in which the first job registered the agent (after the job is executed the agent self destructs)

Azure Pipelines Agent Docker Image

You can build your image to run the jobs (as long you follow the interface) or you can use the provided basic docker images.

If the provided images don't have the necessary software, you need to deploy; you can base your image on the provided basic image. By using the base image, you don't have to implement the logic that registers the agent and deletes the container.

You need to build and publish it to a docker registry (either public or private). The repo contains an Azure Pipeline YAML file that builds and publishes the image to a registry.

Learn how to build and publish the image(s)

Task

This repo contains a task packaged as an Azure DevOps Extension that provisions an Azure Container Instance (based on an image of your choice stored in any docker registry (anonymous or authenticated)).

You can opt to build the extension yourself (and publish on the marketplace as a private extension) or install the publicly available extension from the marketplace.

The task works in symbiosis with the agent docker image to make sure the agent is registered, only runs one job and the container is deleted after the job is finished.

Learn how to use the extension

Publishing the extension

The provided pipeline task is packaged as an Azure DevOps Extension. To publish the extension, you need to have a publisher and change the manifest to include your publisher id.

Read all the necessary steps in Develop a web extension for Azure DevOps Services

Requirements

You have to have a dedicated subnet in your virtual network just to run the ephemeral agents.

The agent must be running in the same Azure location as the virtual network.

The Samples folder has a create-vnet.sh script that creates a virtual network and a bunch of subnets. It creates a resource group called ephemeral-agents-sample with a vnet that holds roughly 64K IP addresses and 5 subnets (arbitrary names and intention just for illustration purposes)

• Compute with ~2K IP addresses for compute resources
• Storage with ~2K IP addresses for storage resources
• data with ~2K IP addresses for data resources
• Bastion with 0.5K IP addresses to have VMs that act as bastions if needed
• Agents with 0.5K IP addresses for ephemeral agents (this is essential agents need to be on a dedicated subnet)

All the subnets share the same network security group.

Recommendations

Use one agent pool per team project/virtual network

Provisioned agents should be placed in its resource group (one per agent pool).

Sample pipeline

There are some sample pipelines that show how you can use it to deploy to a resource in a private vnet:

• [Samples/storage-pipeline/Azure-Pipelines.yml] Copy files to a azure storage container

The pipeline assumes the existence of:

• A virtual network called sampleprivatenetwork in the ephemeral-agents-sample-vnet resource group (create-vnet.sh takes care of that for your)
• An azure storage account called ephemeralsamplestorage with a container called test (access restricted to the created private virtual network. You can use the provided create--storage.sh)
• An Azure Resource Manager service connection called sample-azure-subscription (with permissions to create and access the resource groups)
• An agent pool called sample-ephemeralpool (the pipeline uses system.accesstoken OAuth token, so build account requires enough permissions to register agents on the pool) see more (you can also opt to use a PAT token if you prefer)
• A docker container registry service connection called ephemeralsampleregistry (in this case it is assumed to be an Azure Container Registry but that is mainly an assumption based on the image fully qualified name). See Create a service connection and Docker Registry service connection

You need an azure container registry (you can use for example the provided create-acr.sh script to create one) that holds the image to be used. The pipeline assumes an Azure Container Registry named ephemeralsampleregistry. Since a registry needs to be universally unique, it is most likely you have to change the name of the registry in both the script and the pipeline.

Known issue(s)/Limitations

Azure Resource Manager Service service connections that use certificates are not supported.

Azure Container Instances with virtual networks are currently in preview, and they have some limitations.

See more at Deploy container instances into an Azure virtual network preview limitations

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

azure-pipelines-ephemeral-agents's Issues

Task doesn't show up in search if Ephemeral keyword is used

After the initial install from the Marketplace, using either classical pipelines or yaml it does not work.

With classical pipelines there is a grey "installed" and no tasks available. Any attempts at using it with yaml yields an error.

I've gone over it with Microsoft support and they say the issue is with the extension, and they're unable to help further, although they're trying to see if they can get some backend logs.

Error During Container Instance Creation

Receiving this error during CI creation in the Ephemeral agent step:

2020-10-07T16:42:41.3143033Z ##[error]Error Code: [1]
2020-10-07T16:42:41.3169513Z ##[debug]Processed: ##vso[task.issue type=error;]Error Code: [1]
2020-10-07T16:42:41.3171321Z ##[error]Error: Couldn't create Azure Container Instance
2020-10-07T16:42:41.3172481Z ##[debug]Processed: ##vso[task.issue type=error;]Error: Couldn't create Azure Container Instance
2020-10-07T16:42:41.3173219Z ##[debug]task result: Failed
2020-10-07T16:42:41.3181300Z ##[error]Script failed with error: UnexpectedError: The command failed with an unexpected error. Here is the traceback:
No module named 'azure.mgmt.network.v2018_08_01'
Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-tuw8_17y\knack\cli.py", line 215, in invoke
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-v8nptj_b\azure\cli\core\commands\__init__.py", line 654, in execute
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-v8nptj_b\azure\cli\core\commands\__init__.py", line 718, in _run_jobs_serially
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-v8nptj_b\azure\cli\core\commands\__init__.py", line 711, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-cpn0gwyu\six.py", line 703, in reraise
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-v8nptj_b\azure\cli\core\commands\__init__.py", line 688, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-v8nptj_b\azure\cli\core\commands\__init__.py", line 325, in __call__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-v8nptj_b\azure\cli\core\__init__.py", line 784, in default_command_handler
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-s56abhv8\azure\cli\command_modules\container\custom.py", line 190, in create_container
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-s56abhv8\azure\cli\command_modules\container\custom.py", line 286, in _get_vnet_network_profile
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-unpacked-wheel-v0h_67b1\azure\mgmt\network\_network_management_client.py", line 5336, in subnets
ModuleNotFoundError: No module named 'azure.mgmt.network.v2018_08_01'
To open an issue, please run: 'az feedback'

Pipeline consists of two agents:

I have tried using values in each filed and variables in each field, the same error is the result.

Could not resolve host: dev.azure.com

A few months ago i was able to complete the sample pipeline and it runs with no problems, but a few days ago when i try to run it, get the error:

Could not resolve host: dev.azure.com

The error appears just after "1. Determining matching Azure Pipelines agent...", in the curl command

Error During Container Instance Creation (Windows)

Error Code: [1]
Error: Couldn't create Azure Container Instance
Script failed with error: The requested resource is not available in the location 'westeurope' at this moment. Please retry with a different resource request or in another location. Resource requested: '2' CPU '8' GB memory 'Windows Server 2019 - After 2B' OS virtual network

Eventually timeout on job executions

I'm trying to use Ephemeral Agents to use Terraform. It uses a storage account to keep state of the resources, but, eventually, I receive this error:

I've created the agent with the pipeline task, and assigned to a VNet...

Thank you,

Please support ephemeral agents in K8s

I would like to see the work in AKS, EKS, GKE, and vanilla K8s.