microsoft / azureclusterlesshpc.jl

A Julia package for clusterless distributed computing on Azure

AzureClusterlessHPC.jl - Simplified distributed computing

Overview

AzureClusterlessHPC.jl is a package for simplified parallal computing on Azure. AzureClusterlessHPC.jl borrows the syntax of Julia's Distributed Programming package to easily execute parallel Julia workloads in the cloud using Azure Batch. Instead of a parallel Julia session, users create one or multiple worker pools and remotely execute code on them.

AzureClusterlessHPC provides macros that let us define functions on batch workers, similar to how @everywhere works for a parallel Julia session:

# Define a function
@batchdef hello_world(name)
    print("Hello $name")
    return "Goodbye"
end

We can then either execute this function on our local machine or as a batch job using the @batchexec macro (which is similar to Julia's @spawn macro for parallel Julia sessions):

# Execute function on local machine
out = hello_world("Bob")

# Execute function via Azure Batch
out = @batchexec hello_world("Jane")

Using the pmap function in combination with @batchexec allows us to run a multi-task batch job:

# Execute a multi-task batch job
out = @batchexec pmap(name -> hello_world(name), ["Bob", "Jane"])

Installation

To install AzureClusterlessHPC.jl, run the following command from an interactive Julia session (press the ] key and then type the command):

] add AzureClusterlessHPC.jl

Before being able to use AzureClusterlessHPC.jl you need to create a few Azure resources. Follow the instructions here.

Documentation

Follow this link to the documentation.

Applications

AzureClusterlessHPC can be used to bring various distributed computing applications in Julia to Azure. Check out the notebooks in the examples section to find tutorials for the following applications:

• Generic batch, map-reduce and iterative map-reduce examples

• Deep learning with AzureClusterlessHPC.jl and Flux.jl

• Seismic imaging and inversion with COFII.jl and JUDI.jl

Credits

AzureClusterlessHPC.jl is developed and maintained by the Microsoft Research for Industries (RFI) team.

azureclusterlesshpc.jl's Issues

pmap does not behave the same as in Distributed.jl

The pmap in Julia Distributed has an option on_error which specifies the output when the function on one(s) of the workers catches an error. For example,

julia> pmap(x->sqrt(x-2), 1:3;on_error=ex->100)
3-element Vector{Real}:
 100
   0.0
   1.0

Would it be simple to add this functionality to @batchexec pmap because it is currently not supported?

julia> @batchexec pmap(x->sqrt(x-2), 1:3;on_error=x->100)
ERROR: LoadError: BoundsError: attempt to access 1-element Vector{Any} at index [2]
Stacktrace:
 [1] getindex(A::Vector{Any}, i1::Int64)
   @ Base ./array.jl:801
 [2] create_multi_task_expression_list_pmap(expr::Expr; options::Nothing)
   @ AzureClusterlessHPC ~/.julia/dev/AzureClusterlessHPC/src/core/batch_macros.jl:392
 [3] create_expression_to_submit_batch_job(expr::Expr; options::Nothing)
   @ AzureClusterlessHPC ~/.julia/dev/AzureClusterlessHPC/src/core/batch_macros.jl:483
 [4] create_expression_to_submit_batch_job(expr::Expr)
   @ AzureClusterlessHPC ~/.julia/dev/AzureClusterlessHPC/src/core/batch_macros.jl:482
 [5] var"@batchexec"(__source__::LineNumberNode, __module__::Module, expr::Any)
   @ AzureClusterlessHPC ~/.julia/dev/AzureClusterlessHPC/src/core/batch_macros.jl:716
in expression starting at REPL[6]:1

deploy.sh SUBSCRIPTION_ID

Hi,

nice work! My group is exploring azure/AWS for scientific computing.

Following the installation guide for AzureClusterlessHPC on a fresh Ubuntu 18.04 instance the deploy-script for creating azure batch and storage accounts fails with the error "invalid scope".

As far as I could check the command
az account show --query subscription --output tsv
in line 36 gives nothing. To obtain the subscription id the command
az account show --query id --output tsv
must be used.

Can you comment?

Add `lower` for safety

AzureClusterlessHPC.jl/src/AzureClusterlessHPC.jl

Line 81 in e94af8b

global __container__ = try __params__["_BLOB_CONTAINER"] catch; nothing end # Azure container for temp files

blob desn't accept capital lettersso it may be safer to do lower(container) to make sure it's a valid name.

Connect to DockerHub

Hello again. I've tried using a personal Docker Hub account with a private registry and an image, and I've run into nearly the same issue at #19. Instead, the error message I get is:

pull access denied for myuser/myimage, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

I have done a docker login with credentials for myuser in the terminal before starting Julia, so I'm expecting that to work. Maybe this is the underlying issue of #19 instead.

This is an even closer setup to Azure/Batch#107 because I'm using Dockerhub. The only difference between Azure/Batch#107 and this issue is I'm using AzureClusterlessHPC.jl and I'm using my personal Dockerhub private registry.

Thanks again for your help.

Connect to Azure Container Registry

I've been following the julia_batch_docker.ipynb example, but trying to use Azure Container Registry instead of DockerHub and I've been using the container option in the parameters.json ("_CONTAINER": "myorgregistry.azurecr.io/myimage:v0.1.0") but I continue to get an error ContainerInvalidImage with unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. This is a private org container registry.

This is the closest example that I've found to another user attempting this solution: Azure/Batch#107
And here's some docs for Python code that seems to do what I want: https://docs.microsoft.com/en-us/azure/batch/batch-docker-container-workloads#prefetch-images-from-a-private-container-registry

To me it seems like I need to be using a managed identity and to create batch pools with them. Is this something that can be supported by AzureClusterlessHPC.jl? Or is there a better solution than what I've found?

Thank you for your help!