External GCS bucket setup guide

Bobsled can deliver data to an external bucket, housed within a non-Bobsled GCP project. For example, this could be a bucket in the provider's GCP project or in a consumer's GCP project. This provides maximum flexibility and control over the bucket itself.

Setup instructions

Step 1: Setting up a share to an external bucket in Bobsled

1. On the share page, click the box Pick Destination

2. Choose the cloud platform Google Cloud Storage and choose the region of the target bucket

3. Select "External bucket" and press continue

Step 2: Set up destination access

Bobsled has flexible options for how to write to an external bucket.
Provide the bucket name and optional path to write to:

If desired you can enable:

• Bobsled Share Path to add "share ID" and "latest" to the path written by Bobsled. This should be used when delivering via multiple shares to the same bucket to ensure they don’t overlap.

• Mirror source data to allow Bobsled to delete files that are removed from the source. This mode tells Bobsled to match the contents of the source bucket to the destination.

TIP:
When using a Cloud Data Warehouse source or when removing files is not required, this setting is suggested to be off.

Finally, select how you'd like Bobsled to access the bucket:

• Bobsled Service Account: you give a Bobsled service account sufficient access to the external bucket

• Assume Service account: you give Bobsled the ability to assume a service account with write access to the external bucket

Follow the steps below depending on your preferred access configuration, and once you’ve entered the relevant details, click grant access and then finish.

Access configuration

Prerequisites

• To configure Bobsled access to the bucket, your account must have the sufficient permissions to create policies and assign roles in Google Cloud Storage.

• If your GCP restricts domains that can access resources, you must allow the Bobsled domain.

  • To add the Bobsled domain to your organization policy, you will need the Directory Customer ID from the Bobsled App side panel, click Environment and scroll to Google Cloud Storage setting details

  • Please visit Add domain to organization in GCP ↗ and follow the steps to allow the Bobsled domain.

Option 1: Bobsled Service Account

Assign Permissions to Bobsled Service Account in the Bucket

NOTE:
Before you start this step, you’ll need the Bobsled Service Account Email address from the destination setup modal.

1. Log into the Google Cloud Platform Console as a project editor.

2. From the Home dashboard, choose Cloud Storage > Buckets

3. Find the bucket you want to grant Bobsled Access to and click on the ellipses to the right of the bucket's row. Select Edit Access.

4. Click the Add Principal button.

5. In the New principals field, paste in the Bobsled Service Account Email address and select it from the returned options.

6. Select the Role drop down. Add two roles:

   • Storage Object Creator. If you want Bobsled to be able to delete objects in the external bucket, give Storage Object User instead.

   • Storage Object Viewer

7. Click Save.

Option 2: Assume Service Account

Assign Permissions to Bobsled Service Account in the Bucket

NOTE:
Before you start this step, you’ll need the Bobsled Service Account Email address from the destination setup modal.

1. Log into the Google Cloud Platform Console as a project editor.

2. From the Home dashboard, choose IAM > Service Accounts

3. Find the Service Account that has access to the external bucket to and click on the ellipses to the right and select Manage permissions.

4. Under the principals with access to this service account, click Grant Access

5. In the New principals field, paste in the Bobsled Service Account Email address and select it from the returned options.

6. Select the Role drop down. Add two roles:

   • Service Account Token Creator

   • Service Account User

7. Click Save.

8. Return to the Bobsled App destination modal and provide the Service account for Bobsled to Assume and click Grant access.

9. Click finish.

For KMS Encrypted Buckets Only

Grant Bobsled Service Account permission on Cryptographic Keys:

1. Log in to the GCS Console as a project editor.

2. Navigate to the Home dashboard. Choose Security > Key Management

3. Select the key ring that is assigned to your GCS bucket.

4. Click Show Info Panel in the upper-right corner and click add principal button.

5. In the New principals field, search for the Bobsled service account

6. From the Select a role dropdown, select the Cloud KMS CrytoKey Encryptor/Decryptor role.

7. Click the Save button.

After setting up the destination in a Share, and picking a source, you can get started and create a data transfer to share data with your consumers.