Setup guide

This article will describe the steps for a Provider to grant Bobsled access to an Amazon S3 bucket to be used as a source in Bobsled.

Prerequisites

To configure Bobsled access to the bucket, your account must have the sufficient permissions to create policies and assign roles in Amazon Web Services (AWS).

Setup instructions

Bobsled has created a cloud formation template that can be used to quickly create the role and trust policy required below. This saves time from having to create the policies manually and makes sure the policies are correctly configured. You can either:

• Use Bobsled Quick start AWS Cloud Formation Template or,

• Create the IAM policies and roles manually.

Option 1: Quick start with AWS Cloud Formation Template

Bobsled has created a cloud formation template that can be used to quickly create the role and trust policy required below. This saves time from having to create the policies manually and makes sure the policies are correctly configured.

The path parameter is optional. If you want to filter Bobsled's access to a specific path, fill out the path parameter in the stack without leading or trailing slashes. If you want Bobsled to access the full bucket, leave this parameter empty.

Click on the Quick create stack ↗ in the AWS Console to get started. This link allows you to quickly configure and create the required permission for Bobsled.

Alternatively, you can run the stack creation in the AWS CLI using the command below.

ActionScriptActionScriptaws cloudformation create-stack --stack-name BobsledS3BucketSourceAccessSetup --template-url https://bobsled-cloud-formation-templates.s3.us-east-2.amazonaws.com/S3BucketSourceSetup.yaml --parameters ParameterKey=BucketName,ParameterValue=your-bucket-name ParameterKey=BobsledReadAccessRoleArn,ParameterValue=bobsled-role-from-app ParameterKey=BobsledReadExternalId,ParameterValue=bobsled-external-id-from-app ParameterKey=RoleName,ParameterValue=BobsledAccessRole ParameterKey=TrustPolicyName,ParameterValue=BobsledTrustPolicy ParameterKey=Path,ParameterValue=optional-path --region us-east-2 --capabilities CAPABILITY_NAMED_IAM

To complete the setup of the Source in the next step to configure your Amazon S3 Source, you will need to obtain the ARN of the role created by the stack.

You can access this information using the following CLI command. Ensure that the stack name matches the name you used. 

ActionScriptActionScriptaws cloudformation describe-stacks --stack-name BobsledS3BucketSourceAccessSetup

Alternatively, you can find the output role ARN by locating the stack in the CloudFormation console ↗. You can also see the stack template which includes the details on the policy and role that will be created by accessing the template (.yaml) ↗

Option 2: Grant access to your bucket

Step 1: Create an IAM Policy

1. Login to AWS Management Console;

2. From the ‘Services’ dropdown, select IAM under ‘Security, Identify & Compliance’ section;

3. Click Account Settings on the left-hand panel;

4. Expand the ‘Security Token Service (STS)’ list, find the AWS region corresponding to the region where your bucket is located, and choose ‘Activate’ if the status is ‘Inactive’;

5. Choose Policies ↗ from the left-hand navigation pane;

6. Click Create Policy;

7. Click the JSON tab;

8. Add JSON policy that allows Bobsled to read from the S3 bucket. The following policies provide Bobsled with required permissions to read data from a specified list of entire buckets or subfolders. Please replace placeholders with your bucket name(s), and pay special attention to the trailing /*. It should be present on the first statement, but not on the second one;

   1. Allow Bobsled to read from an entire bucket;

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "s3:GetObject",
              "s3:GetObjectVersion"
            ],
            "Resource": [
              "arn:aws:s3:::<bucket-name>/*",
              "arn:aws:s3:::<bucket-name>/*"
            ]
          },
          {
            "Effect": "Allow",
            "Action": [
              "s3:ListBucket",
              "s3:GetBucketLocation"
            ],
            "Resource": [
              "arn:aws:s3:::<bucket-name>",
              "arn:aws:s3:::<bucket-name>"
            ]
          }
        ]
      }

   2. Allow Bobsled to read from subfolders within a bucket. 

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "s3:GetObject",
              "s3:GetObjectVersion"
            ],
            "Resource": [
              "arn:aws:s3:::<bucket-name>/*",
              "arn:aws:s3:::<bucket-name>/*"
            ]
          },
          {
            "Sid": "AllowListingOfSubFolder",
            "Action": [
              "s3:ListBucket",
              "s3:GetBucketLocation"
            ],
            "Effect": "Allow",
            "Resource": [
              "arn:aws:s3:::<bucket-name>"
            ],
            "Condition": {
              "StringLike": {
                "s3:prefix": [
                  "subfolder/*"
                ]
              }
            }
          }
        ]
      }

9. Click Next: Tags;

10. Optionally add tags to the policy to help identify, organize, or search for AWS resources;

11. Create a policy name (e.g. bobsled_access), and optionally add a description;

12. Click create policy.

Step 2: Create an IAM Role

1. Login to AWS Management Console;

2. From the ‘Services’ dropdown, select IAM under the ‘Security, Identity & Compliance’ header';

3. Click Roles on the left-hand panel;

4. Click the Create role button;

5. Under ‘Trusted entity type, select Custom trust policy

6. Set the trust policy using the following JSON:

   • Replace the <awsBobsledReadArn> and <awsBobsledReadExternalId> in the JSON below with the values found in the Bobsled application. These values can be found by visiting Data Sources > Add Source > select Amazon S3.
     
     

     

     {
       "Version": "2012-10-17",
       "Statement": [
         {
           "Effect": "Allow",
           "Principal": {
             "AWS": "<awsBobsledReadArn>"
           },
           "Action": "sts:AssumeRole",
           "Condition": {
             "StringEquals": {
               "sts:ExternalId": "<awsBobsledReadExternalId>"
             }
           }
         }
       ]
     }

     
     

7. Click the Next button;

8. Find the policy you created in the previous section, and select this policy;

9. Click the Next button;

10. Enter a name and description for the role, and click the Create role button;

11. Record the Role ARN value located on the role summary page. You will use the Role ARN to configure your source in Bobsled.

Step 3: For KMS Encrypted Buckets Only

Grant Bobsled Role Access to Encryption Keys:

1. Navigate to the S3 bucket and click on Properties > Default encryption section.

2. Click the link for the Encryption key ARN

3. Copy the Encryption Key ARN and note it separately.

4. Click Edit to edit the policy

   1. You will need to add the following inline to your key policy, replacing <awsBobsledReadArn> with the ARN Supplied in the Bobsled UI, and <createdRoleARN> with the ARN of the role you created for Bobsled:

      {
                  "Sid": "Allow use of the key",
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": [
                          "<awsBobsledReadArn>"
                      ]
                  },
                  "Action": [
                      "kms:Encrypt",
                      "kms:Decrypt",
                      "kms:ReEncrypt*",
                      "kms:GenerateDataKey*",
                      "kms:DescribeKey"
                  ],
                  "Resource": "*"
              },
              {
                  "Sid": "Allow attachment of persistent resources",
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": [
                          "<createdRoleARN>"
                      ]
                  },
                  "Action": [
                      "kms:CreateGrant",
                      "kms:ListGrants",
                      "kms:RevokeGrant"
                  ],
                  "Resource": "*",
                  "Condition": {
                      "Bool": {
                          "kms:GrantIsForAWSResource": "true"
                      }
                  }
              }

5. Navigate to the Policy your created in Step 1.

6. Add the following inline to the policy, replacing <kms_key_arn> with the ARN of the S3 bucket key.:

 {
            "Effect": "Allow",
            "Action": [
                "kms:decrypt",
                "kms:encrypt"
            ],
            "Resource": "<kms_key_arn>"
        }

Last step: Finish source configuration in Bobsled

Once you have created the relevant roles and access policies, finish granting Bobsled access via the Bobsled app:

1. From the menu on the left, select Data Sources.

2. Click Add Source

3. Enter a human readable name to describe the source.

4. Select Amazon S3 and the region of your source bucket.

5. Click Continue to move to the next step.
   
   

   

   
   

6. Enter the Read-only ARN that you have created in the AWS Console.

7. Enter the name of the bucket. (Optionally, you may also include the path to the root of your source.)

8. Click Save.

Once you’ve successfully configured your Source, you can add it to a Share, choose a Destination, and a create a transfer to start sharing your data.