Consuming a data transfer in Azure Blob Storage

This article will describe how to consume a Bobsled data transfer in Azure Blob Storage. This describes the process to access the data delivered to the Bobsled-managed destination using the Bobsled-managed application and/or Consumer-managed application.

This document provides three simple examples of how to access a data transfer on your computer’s command-line interface. This method is ideal to view, copy and sync the data transfer from the Bobsled-managed container to your own container. However, a data transfer can be accessed and consumed in production from any Azure client with the account that has been granted access to the share.

Prerequisites

• Before consuming a data transfer, a data transfer must be sent to the Azure Blob Storage destination.

• An access method must be configured in Bobsled.

  • Bobsled Managed Application: a secret must be generated in Bobsled to grant access to the Bobsled Managed Application used to access the data.

  • Consumer Managed Application: an Azure Multi-tenant application must be configured access in Bobsled. In order to consume the data via this method, you must know your application's access credentials.

• Azure Blob Storage data can only be accessed via the Azure Blob Storage command-line tools (Azure CLI or AzCopy) or the Azure REST API. For help installing the Azure CLI tools, visit Installing the AzCopy & Azure CLI.

Consuming a data transfer

1. From the Shares list page, click on the share that you would like to access.

2. Once a data transfer in the share has been completed, select the button Access Data.
   
   

   

   
   

There are two main ways to consume your transfer via the access data;

• AzCopy,

• Azure CLI (Command line)

Option 1: Accessing via AzCopy

Using the Azure Blob Storage command-line tool, you can list, copy, and sync the contents of the data transfer in Azure Blob Storage. To use the following commands, you will need to copy the Storage Container URI located in the access data dialog as pictured above. All of the commands described below are provided within the access data dialog.

Step 1: Login to the CLI

1. Set your client secret to the environment variable "AZCOPY_SPA_CLIENT_SECRET"

   • Mac/Linux command:
     

     export AZCOPY_SPA_CLIENT_SECRET=<client-secret>

   • Windows command prompt:
     

     set AZCOPY_SPA_CLIENT_SECRET=<client-secret>

   • PowerShell:
     

     $Env:AZCOPY_SPA_CLIENT_SECRET=""

     
     

2. Run the Login command. The application-id (Bobsled Managed or Consumer Managed) and tenant ID (Bobsled Tenant ID) are provided in the access data dialog.
   

   azcopy login --service-principal --application-id <application-ID> --tenant-id <Bobsled tenant-ID>

3. Output:

   

   
   
   

Step 2: List the contents

1. To list the blobs in storage container, you will use the command "azcopy list". For more information, visit Azure documentation ↗
   

   azcopy list <storage-container-URI>

2. Output:

   

   
   
   

Step 3: Generate SAS Token

NOTE:
You can will need to use the Azure CLI to generate the SAS token. After generating it, Bobsled recommends copy and syncing using AzCopy.

1. Login to Azure CLI. The application-id (Bobsled Managed or Consumer Managed) and tenant ID (Bobsled Tenant ID) are provided in the access data dialog.
   

   az login --service-principal -u <application-id> -p <client secret> --tenant <Bobsled tenant-id>

2. Generate SAS Token. You can find the storage account name and container name from the Storage Container URI: https://<storage account name>.blob.core.windows.net/<container name>/<data transfer>
   

   az storage container generate-sas --account-name <storage-account-name> --name <container-name> --permissions lr --expiry <expiry date> --auth-mode login --as-user

Step 4: Switch to your Tenant to Copy and Sync the data

• For Bobsled-managed Application:
  

  azcopy login --tenant-id <your-tenant-id>

• Consumer managed Application:
  

  azcopy login --service-principal --application-id <application-id> --tenant-id <your-tenant-id>

NOTE:
Optional for data transferred to Consumer-managed applications: Bobsled recommends logging in using the same application credentials as configured in Bobsled of use. Use the tenant ID where the application was created.

Step 5: Copy the contents

To copy the blobs in the Bobsled storage container to your target storage container, you will use the command "azcopy copy". For more information, visit: Azure’s documentation ↗

azcopy copy "<storage-container-URI?[SAS]>" "<target-container-URI>"

TIP:
Append the SAS Token created to the end of the Bobsled Storage Container URI. You must include a "?" in front of the token.

Step 6: Sync the contents

To sync blobs from the Bobsled storage container to your target storage container and only copy modified blobs from the source container, you will use the command "azcopy sync". For more information, visit Azure documentation ↗

azcopy sync "<storage-container-URI?[SAS]>" "<target-container-URI>"

Output:

Option 2: Accessing Data via Azure CLI

Step 1: Login to the CLI

az login --service-principal -u <application-ID> -p <client-secret> --tenant <Bobsled-tenant-ID>

Step 2: List the Contents of the Storage Container

To list the contents of the storage container, you will use the "az storage blob list" command. For more information visit Azure documentation ↗

az storage blob list --auth-mode login --blob-endpoint <blob-endpoint> -c <container-name> --delimiter '/'

When using the Azure CLI, you will use three important parameters: --blob-endpoint, --container name, and --delimiter. The parameters can be derived from the provided Storage Container URI: https://<storage account name>.blob.core.windows.net/<container name>/<data transfer>

• blob-endpoint = https://<storage account name>.blob.core.windows.net

• Container name = <container name>

• Delimiter = <data transfer>
  

Please use AzCopy to copy and sync data from the Bobsled storage container to your target storage container.