Accessing Data in GCS

This article explains how to access data delivered to a Google Cloud Storage destination managed by Bobsled. The examples below demonstrate how to use a command-line interface (CLI) to browse, copy, or sync files from the Bobsled-managed bucket to a bucket you control.

While the CLI is used in these examples, data can also be accessed programmatically or integrated into production workflows using any GCS-compatible tool—provided the requesting account has been granted the appropriate access.

Prerequisites

• In Bobsled Transfers, before consuming a data transfer, a data transfer must be sent to the destination, and access must be configured in Bobsled for the identity that is consuming the data. To learn how to configure access to the destination, please visit Configure a Google Cloud Storage Destination

• In Bobsled Sledhouse, at least one Data Product must have been shared to the destination.

• If you are accessing the data via the GCP command-line tool(s), you must install the CLI. Visit Installing the GCP CLI for more information.

Consuming a data transfer via a Share in Bobsled Transfers

1. From the Shares list page, click on the share that you would like to access.

2. Once a data transfer in the share has been completed, select the button Access Data.

Option 1: Accessing Data via Web Console

You can easily access the data using the web console link to view and download the data. To do so, please be sure to log in to the [insert cloud’s console name] with the account that has been configured to access the Bobsled share.

1. Select the Web console tab in the access dialog

2. Click the link icon to view the data in the GCP Web Console.

Option 2: Accessing Data via Command line

Using the Google Cloud Storage command-line tools (Gsutil or Gcloud), you can list, copy, and sync the contents of the data transfer in Google Cloud Storage. To use the following commands, you will need to copy the Cloud Storage URI located in the access data dialog as pictured above.

Log in to the GCP Command-line

1. Run the login command.
   

   gcloud auth login

   • If you would like to access the data as a service account, run the command:
     

     gcloud auth activate-service-account [service-account-email] --key-file=[path-to-private-key-file]

Access via Gcloud command-line tool

NOTE:
To use the following commands, you will need to copy the Cloud Storage bucket URI located in the access dialog as pictured above.

1. List the contents. To list the data in the bucket, you will use the command gcloud ls ↗
   

   gcloud storage ls -r <storage-bucket-URI>

   • Parameters used: -r (recursive) lists all objects in a bucket

   • Optional parameters to use with the list command: -l (info) additional information about the bucket (object size, creation time, etc)

2. Copy the contents to your bucket. To copy the data in the bucket, you will use the command gcloud cp ↗
   

   gcloud storage cp -r <storage-bucket-URI> <your bucket/path>

   • Parameters used: '-r' (recursive) copies entire directory tree

   • Optional parameters to use with the copy command: '-n' (no-clobber) prevent overwriting the content of existing files at the destination.

3. Sync the contents. Please use gsutil to sync the contents of the data transfer to your bucket.

Access via Gsutil command-line tool

1. List the contents. To list the data in the bucket, you will use the command gsutil ls ↗
   

   gsutil ls -r <storage-bucket-URI>

   • Parameters used: -r (recursive) lists all objects in a bucket

   • Optional parameters to use with the list command: -l (info) additional information about the bucket (object size, creation time, etc)

2. Copy the contents to your own bucket. To copy the data in the bucket, you will use the command gsutil cp ↗
   

   gsutil cp -r <storage-bucket-URI> <your bucket/path>

   • Parameters used: '-r' (recursive) copies entire directory tree

   • Optional parameters to use with the copy command: '-n' (no-clobber) prevent overwriting the content of existing files at the destination

3. Sync the contents. Use sync if you would like to copy only files that are new or updated. Learn more about the command rsync ↗
   

   gsutil rsync <storage-bucket-URI> <your bucket/path>

   • Parameters used: '-r' (recursive) copies entire directory tree.
     

Consuming a Data Product via Data Fulfillment in Sledhouse

FEATURE IN PUBLIC PREVIEW:
• While in Public Preview, expect some limitations. Your feedback is crucial as we build the product.
• We look forward to your feedback and to helping you integrate Sledhouse into your data architecture. Feel free to reach out if you have any questions about setup, configuration, or advanced use cases.

1. From the Data Fulfillment list page, click on the Data Consumer whose that you would like to see the access details.

2. Once a data share has been completed, select the button Access Data.

3. Select the copy icon and send it to your data customer. Additionally, you can follow the steps relayed in Bobsled Transfers details.

NOTE:
• Using Bobsled Sledhouse Data Fulfillment functionality, Bobsled will provide the discrete paths for any individual Data Products rather than a single bucket.