Consuming a data transfer in Amazon S3

This article will describe how to consume a Bobsled data transfer in Amazon S3. This describes the process to access the data delivered to the Bobsled-managed destination.

This document provides three simple examples of how to access a data transfer via a data consumer's command-line interface. This method is ideal to view, copy and sync the data transfer from the Bobsled-managed bucket to a data consumer's bucket. However, a data transfer can be accessed and consumed in production from any Amazon S3 client with the account that has been granted access to the share.

Prerequisites

Before consuming a data transfer:

• A data transfer must be sent to the Amazon S3 destination and,

• Access must be configured in Bobsled for the identity that is consuming the data. Learn more how to configure an Amazon S3 destination.

• If you are accessing the data via the AWS command-line tool, you must install the CLI.

NOTE:
Please ensure that the consumer’s IAM policy includes the necessary permissions for listing and reading from an S3 bucket. If the policy requires specifying buckets that a user/role can access, the destination bucket ARN and access point ARN should be included as a resource in the consumer’s policy that at the least, has the following permissions: 

• s3:GetObject: Read objects from the bucket.

• s3:ListBucket: List objects within the bucket.

• s3:GetBucketLocation: determine the geographical location of an S3 bucket.

  Learn more about help finding the destination bucket ARN in Amazon S3 ↗

Sample Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowListBucketAndGetLocation",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "{S3_BUCKET_ARN}",
                "{ACCESS_POINT_ARN}"
            ]
        },
        {
            "Sid": "AllowGetObject",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAttributes",
                "s3:GetObjectVersion",
                "s3:GetObjectTagging"
            ],
            "Resource": [
                "{S3_BUCKET_ARN}/*",
                "{ACCESS_POINT_ARN}/object/*"
            ]
        }
    ]
}

• The bucket and access point ARNs can be found in the UI for a given share by selecting Access Data → Advanced

Consuming a data transfer

1. From the Shares list page, click on the share that you would like to access.

2. Once a data transfer in the share has been completed, select the button Access Data.

3. Select the copy icon and send to your data customer.
   

Option 1: Accessing Data via Web Console

Consumer's can easily access the data using the web console link to view and download the data.

1. Select the Web console tab in the access dialog.

2. Copy the AWS Web Console link icon and send to your customer. They will be prompted to log in to their AWS console if they are not already.

Option 2: Accessing Data via Command line

Using the AWS command-line tool, consumers can list, copy, and sync the contents of the data transfer in Amazon S3. Bobsled provides out of the box list, copy, and sync commands tailored to the data consumer's bucket.

TIP:
For the following commands, please ensure that all S3 URIs (Bobsled URL and your bucket path) are enclosed in quote (‘‘) if any spaces in the URI exist.

Step 1: Log In to the CLI

1. Run the command aws configure. You will be prompted on the cli to enter your AWS Access Key ID, AWS Secret Access Key, default region, and output format.

   • For the default region, be sure that it is set the same region as in the Bobsled share.

   • Set the output format to json
     
     

     aws configure
     AWS Access Key ID [********************]:
     AWS Secret Access Key [********************]:
     Default region name [us-east-1]:
     Default output format [json]:

Step 2: List the contents

Please visit AWS CLI ↗ for for more information.

• Additional parameters to use with the list command:

  • --recursive (boolean) Command is performed on all files or objects under the specified directory or prefix

  • --human-readable (boolean) Display file sizes in human readable format

  • --summarize(boolean) Displays summary information such as number of objects, total size, etc.
    
    

    aws s3 ls <Bobsled URL>

    
    

    

    
    

    aws s3 ls <Bobsled URL> --recursive --human-readable --summarize

    
    

    

    
    

Step 3: Copy the contents to your own bucket

Please visit AWS CLI ↗ for more information.

NOTE:
To copy data into your own S3 bucket, the bucket must be in the same region as the Bobsled-managed bucket. If the region is not the same, you will receive an ‘Access Denied’ error.

• Copy an individual file:
  

  aws s3 cp < Bobsled URL/<path to file> s3://<your bucket or path>

  

  
  

• Copy all files: --recursive (boolean) Command is performed on all files or objects under the specified directory or prefix. Visit AWS CLI cp command document linked above for more information..
  

  aws s3 cp <Bobsled URL> s3://<your bucket or path> --recursive

  
  

  

Step 4: Sync the contents

Use sync if you would like to copy only files that have are new or updated. Please visit AWS CLI ↗ for more information.

aws s3 sync <Bobsled URL> s3://<your bucket or path>