AzureRMR provides the ability to parallelise communicating with Azure by utilising a pool of R processes in the background. This often leads to major speedups in scenarios like downloading large numbers of small files, or working with a cluster of virtual machines. This is intended for use by packages that extend AzureRMR (and was originally implemented as part of the AzureStor package), but can also be called directly by the end-user.
This functionality was originally implemented independently in the AzureStor and AzureVM packages, but has now been moved into AzureRMR. This removes the code duplication, and also makes it available for other packages that may benefit.
A small API consisting of the following functions is currently provided for managing the pool. They pass their arguments down to the corresponding functions in the parallel package.
init_poolinitialises the pool, creating it if necessary. The pool is created by calling
parallel::makeClusterwith the pool size and any additional arguments. If
init_poolis called and the current pool is smaller than
size, it is resized.
delete_poolshuts down the background processes and deletes the pool.
pool_existschecks for the existence of the pool, returning a TRUE/FALSE value.
pool_sizereturns the size of the pool, or zero if the pool does not exist.
pool_exportexports variables to the pool nodes. It calls
parallel::clusterExportwith the given arguments.
pool_mapcarry out work on the pool. They call
parallel::clusterMapwith the given arguments.
pool_evalqexecute code on the pool nodes. They call
parallel::clusterEvalQwith the given arguments.
The pool is persistent for the session or until terminated by
delete_pool. You should initialise the pool by calling
init_pool before running any code on it. This restores the original state of the pool nodes by removing any objects that may be in memory, and resetting the working directory to the master working directory.
The pool is a shared resource, and so packages that make use of it should not assume that they have sole control over its state. In particular, just because the pool exists at the end of one call doesn’t mean it will still exist at the time of a subsequent call.
Here is a simple example that shows how to initialise the pool, and then execute code on it.
Here is a more realistic example using the AzureStor package. We create a connection to an Azure storage account, and then upload a number of files in parallel to a blob container. This is basically what the
storage_multiupload function does under the hood.
init_pool() library(AzureStor) endp <- storage_endpoint("https://mystorageacct.blob.core.windows.net", key="key") cont <- storage_container(endp, "container") src_files <- c("file1.txt", "file2.txt", "file3.txt") dest_files <- src_files pool_export("cont") pool_map( function(src, dest) AzureStor::storage_upload(cont, src, dest), src=src_files, dest=dest_files )