ADLS, file and block blob uploads gain the option to compute and store the MD5 hash of the uploaded file, via the put_md5 argument to upload_adls_file, upload_azure_file and upload_blob.
Similarly, downloads gain the option to verify the integrity of the downloaded file using the MD5 hash, via the check_md5 argument to download_adls_file, download_azure_file and download_blob. This requires that the file’s Content-MD5 property is set.
Add support for uploading to append blobs, which are a type of blob optimized for append operations. They are useful for data that is constantly growing, but should not be modified once written, such as server logs. See ?upload_blob for more details.
Add support for the Azurite and Azure SDK storage emulators. To connect to the endpoint, use the service-specific functions blob_endpoint and queue_endpoint (the latter from the AzureQstor package), passing the full URL including the account name: blob_endpoint("http://127.0.0.1:10000/myaccount", key="mykey"). The warning about an unrecognised endpoint can be ignored. See the linked pages for full details on how to authenticate to the emulator.
Note that the Azure SDK emulator is no longer being actively developed; it’s recommended to use Azurite.
Add a 10-second fuzz factor to the default starting datetime for a generated SAS, to allow for differences in clocks.
More fixes to the directory handling of list_blobs().
All uploads now include the Content-MD5 header in the HTTP requests, as an error-checking mechanism.
Change maintainer email address.
?blob for more details.
list_blobs, thanks to @cantpitch.list_blobs.create_blob_dir and delete_blob_dir.list_storage_containers and related methods will now check for a continuation marker to avoid returning prematurely (thanks to @StatKalli for reporting and providing a fix).get_account_sas S3 generic, with methods for az_storage resource objects and client endpoints. The az_storage$get_account_sas R6 method now simply calls the S3 method.get_user_delegation_key and get_user_delegation_sas, with methods for az_storage objects and blob endpoints. Similar R6 methods added for az_storage objects.revoke_user_delegation_keys generic and methods to invalidate all user delegation keys for a storage account (and all SAS’s generated with those keys).?sas for more information.silent argument for call_azcopy now uses the azure_storage_azcopy_silent system option for its default value, falling back to FALSE if this is unset.sign_request as a generic, dispatching on the endpoint class. This is to allow for the fact that table storage uses a different signing scheme to the other storage services.list_adls_files that could result in an error with large numbers of files.timeout argument to call_storage_endpoint that sets the number of seconds to wait for an API call.call_azcopy now uses the processx package under the hood, which is a powerful and flexible framework for running external programs from R. The interface is slightly changed: rather than taking the entire commandline as a single string, call_azcopy now expects each AzCopy commandline option to be an individual argument. See ?call_azcopy for examples of the new interface.storage_file_exists generic to check for file existence, which dispatches to blob_exists, azure_file_exists and adls_file_exists for the individual storage types.call_storage_endpoint; this fixes a bug where the token could expire during a long transfer.list_blobs and list_adls_files, check that a field exists before trying to modify it (works around problem of possibly inconsistent response from the endpoint).? (as generated by the Azure Portal and Storage Explorer) to the client functions.multidownload_blob.az_storage$get_*_endpoint() methods now support passing an AAD token for authentication.list_azure_files is now the root, mirroring the behaviour for blobs and ADLSgen2.list_azure_files now includes the full path as part of the file/directory name.recursive argument to list_azure_files, create_azure_dir and delete_azure_dir for recursing through subdirectories. Like with file transfers, for Azure file storage this can be slow, so try to use a non-recursive solution where possible.list_adls_files, list_blobs and list_azure_files more consistent. The first 2 columns for a data frame output are now always name and size; the size of a directory is NA. The 3rd column for non-blobs is isdir which is TRUE/FALSE depending on whether the object is a directory or file. Any additional columns remain storage type-specific.get_storage_metadata and set_storage_metadata methods for managing user-specified properties (metadata) for objects.get_storage_properties rather than having specific functions for blobs, files and directories.file_endpoint, blob_endpoint, adls_endpoint) with an invalid URL will now warn, instead of throwing an error. This enables using tools like Azurite, which use a local address as the endpoint. Calling storage_endpoint with an invalid URL will still throw an error, as the function has no way of telling which storage service is required.do_container_op and call_storage_endpoint to allow direct calls to the storage account endpoint.copy_url_to_storage and multicopy_url_to_storage.NULL download destination will now actually return a raw vector as opposed to a connection, matching what the documentation says.utils::askYesNo; as a side-effect, Windows users who are using RGUI.exe will see a popup dialog box instead of a message in the terminal.copy_url_to_blob function, for directly copying a HTTP[S] URL to blob storage. The corresponding generic is copy_url_to_storage, with a method for blob containers (only).options(azure_storage_retries=N) where N >= 0. Setting this option to zero disables retrying.options(azure_storage_progress_bar=TRUE|FALSE).options(azure_dl_progress_bar=TRUE|FALSE).upload_to_url/download_from_url bugs introduced in last update.src a rawConnection or textConnection object. For downloading, if dest is NULL, the downloaded data is returned as a raw vector, or if dest is a rawConnection, in the connection object. See the examples in the documentation.use_azcopy=TRUE in any upload or download function to call AzCopy rather than relying on internal R code. The call_azcopy function also allows you to run AzCopy with arbitrary arguments. Requires AzCopy version 10.storage_container, create_storage_container, delete_storage_container, list_storage_containers for managing containers (blob containers, file shares, ADLSgen2 filesystems)storage_upload, storage_download, storage_multiupload, storage_multidownload for file transferslist_storage_files, create_storage_dir, delete_storage_dir, delete_storage_file for managing objects within a containerupload_to_url and download_from_url.upload_azure_file to 4MB, the maximum permitted by the API (#5).