Accessing data in Azure Blob Storage

This article explains how to access data delivered to an Azure Blob Storage destination managed by Bobsled. Depending on how access has been configured, you may use either a Bobsled-managed application or your Azure-managed application.

The examples below demonstrate how to access the data using a command-line interface (CLI). This method is ideal for browsing, copying, or syncing files from the Bobsled-managed container to your own Azure container. While the CLI is used here for simplicity, data can also be accessed in production using any Azure-compatible client, as long as the requesting account has been granted access to the destination.

Prerequisites

• In Bobsled Transfers, before consuming a data transfer, a data transfer must be sent to the destination, and access must be configured in Bobsled for the identity that is consuming the data.

  • An access method must be configured in the Share:

    • Bobsled-Managed Application: a secret must be generated in BobSled to grant access to the Bobsled Managed Application used to access the data.

    • Consumer-Managed Application: an Azure Multi-tenant application must be configured for access in Bobsled. In order to consume the data via this method, you must know your application's access credentials.

• In Bobsled Sledhouse, at least one Data Product must have been shared with the destination set up for a Data Consumer.

• Azure Blob Storage data can only be accessed via the Azure Blob Storage command-line tools (Azure CLI or AzCopy) or the Azure REST API. For help installing the Azure CLI tools, visit Installing the AzCopy & Azure CLI.

Consuming a data transfer via a Share in Bobsled Transfer

1. From the Shares list page, click on the share that you would like to access.

2. Once a data transfer in the share has been completed, select the button Access Data.
   
   

   

   
   

There are two main ways to consume your transfer via the access data;

• AzCopy,

• Azure CLI (Command line)

Option 1: Accessing via AzCopy

Using the Azure Blob Storage command-line tool, you can list, copy, and sync the contents of the data transfer in Azure Blob Storage. To use the following commands, you will need to copy the Storage Container URI located in the access data dialog as pictured above. All of the commands described below are provided within the access data dialog.

Step 1: Login to the CLI

1. Set your client secret to the environment variable "AZCOPY_SPA_CLIENT_SECRET"

   • Mac/Linux command:
     

     export AZCOPY_SPA_CLIENT_SECRET=<client-secret>

   • Windows command prompt:
     

     set AZCOPY_SPA_CLIENT_SECRET=<client-secret>

   • PowerShell:
     

     $Env:AZCOPY_SPA_CLIENT_SECRET=""

     
     

2. Run the Login command. The application-id (Bobsled Managed or Consumer Managed) and tenant ID (Bobsled Tenant ID) are provided in the access data dialog.
   

   azcopy login --service-principal --application-id <application-ID> --tenant-id <Bobsled tenant-ID>

   
   

Step 2: List the contents

1. To list the blobs in storage container, you will use the command "azcopy list". For more information, visit Azure documentation ↗
   

   azcopy list <storage-container-URI>

   
   

Step 3: Generate SAS Token

NOTE:
You can will need to use the Azure CLI to generate the SAS token. After generating it, Bobsled recommends copy and syncing using AzCopy.

1. Login to Azure CLI. The application-id (Bobsled Managed or Consumer Managed) and tenant ID (Bobsled Tenant ID) are provided in the access data dialog.
   

   az login --service-principal -u <application-id> -p <client secret> --tenant <Bobsled tenant-id>

2. Generate SAS Token. You can find the storage account name and container name from the Storage Container URI: https://<storage account name>.blob.core.windows.net/<container name>/<data transfer>
   

   az storage container generate-sas --account-name <storage-account-name> --name <container-name> --permissions lr --expiry <expiry date> --auth-mode login --as-user

Step 4: Switch to your Tenant to Copy and Sync the data

• For Bobsled-managed Application:
  

  azcopy login --tenant-id <your-tenant-id>

• Consumer-managed Application:
  

  azcopy login --service-principal --application-id <application-id> --tenant-id <your-tenant-id>

NOTE:
Optional for data transferred to Consumer-managed applications: Bobsled recommends logging in using the same application credentials as configured in Bobsled of use. Use the tenant ID where the application was created.

Step 5: Copy the contents

To copy the blobs in the Bobsled storage container to your target storage container, you will use the command "azcopy copy". For more information, visit: Azure’s documentation ↗

azcopy copy "<storage-container-URI?[SAS]>" "<target-container-URI>"

TIP:
Append the SAS Token created to the end of the Bobsled Storage Container URI. You must include a "?" in front of the token.

Step 6: Sync the contents

To sync blobs from the Bobsled storage container to your target storage container and only copy modified blobs from the source container, you will use the command azcopy sync. For more information, visit Azure documentation ↗

azcopy sync "<storage-container-URI?[SAS]>" "<target-container-URI>"

Option 2: Accessing Data via Azure CLI

Step 1: Login to the CLI

az login --service-principal -u <application-ID> -p <client-secret> --tenant <Bobsled-tenant-ID>

Step 2: List the Contents of the Storage Container

To list the contents of the storage container, you will use the "az storage blob list" command. For more information, visit Azure documentation ↗

az storage blob list --auth-mode login --blob-endpoint <blob-endpoint> -c <container-name> --delimiter '/'

When using the Azure CLI, you will use three important parameters: --blob-endpoint, --container name, and --delimiter. The parameters can be derived from the provided Storage Container URI: https://<storage account name>.blob.core.windows.net/<container name>/<data transfer>

• blob-endpoint = https://<storage account name>.blob.core.windows.net

• Container name = <container name>

• Delimiter = <data transfer>

Please use AzCopy to copy and sync data from the Bobsled storage container to your target storage container.

Consuming a Data Product via Data Fulfillment in Sledhouse

FEATURE IN PUBLIC PREVIEW:
• While in Public Preview, expect some limitations. Your feedback is crucial as we build the product.
• We look forward to your feedback and to helping you integrate Sledhouse into your data architecture. Feel free to reach out if you have any questions about setup, configuration, or advanced use cases.

1. From the Data Fulfillment list page, click on the Data Consumer whose that you would like to see the access details.

2. Once a data share has been completed, select the button Access Data.

3. Copy the Application (client) ID, client secret, and Directory (tenant ID); the data products’ paths and share it with your Data Consumer. Additionally, you can follow the steps relayed in the Bobsled Transfers section and,

NOTE:
• Using Bobsled Sledhouse Data Fulfillment functionality, Bobsled will provide the discrete paths for any individual Data Products rather than a single bucket.