Import FASTQ files from Amazon s3 to process and analyze in Pluto
🔒 Note: To configure an AWS S3 bucket for import into Pluto, you'll need sufficient access to create new users in your AWS IAM console. Contact your database / IT admin for assistance.
Large files such as FASTQ files are commonly shared by vendors in s3 buckets, or may be stored in your company's own S3 buckets for internal user. To avoid having to download those large files and upload them into Pluto, you can grant Pluto limited access to fetch the files out of S3 directly.
This guide will walk you through how to grant Pluto read-only access to a limited set of buckets that you select. If you have any questions at all, please feel free to reach out at security@pluto.bio before proceeding.
Create an AWS S3 bucket
If you haven't already done so, create an S3 bucket in AWS. Move all data to be imported for a single Pluto experiment into that bucket (e.g. all FASTQ files generated in one RNA-seq experiment). Make a note of the bucket name for later.
In the example above, the bucket name is geo-import-gse120284.
Create an User / Access Key in AWS IAM
Create an IAM User
Navigate to IAM > Access Management > Users, and select Add users in the upper right corner.
Set the user name to pluto_bio or something similarly meaningful to you for future reference, and select the Access key - programmatic access option for access type.
On the next step, grant the user permissions with the Attach existing policies directly option. If you haven't yet created a Pluto/s3 access policy, click Create policy.
This will open a new browser tab with a page for creating a policy or set of permissions in AWS. Switch to the JSON section of the Create Policy screen and paste in the following data, changing BUCKET_NAME to the name of the bucket you created.
Nothing else in the JSON needs to be changed except the bucket name(s). Note that each bucket needs 2 string values in the Resource field, one with just the bucket name and a second with the bucket name + /* . The latter allows Pluto to read the objects inside the bucket.
For example, to grant Pluto access to multiple buckets called BUCKET1 and BUCKET2, you would follow this pattern:
Continue to the 3rd step of policy creation, giving the new policy a meaningful name and description, then click Create policy.
Return to the page where you were creating a new user, click the 🔄 icon and search for your newly created policy. Select the policy with the checkbox to its left, and continue to step 4 where you'll click Create user to complete the process.
On the final screen, you'll be presented the option to download the .csv containing access credentials for your newly created user. Remember, these credentials only give the user read-only access to the bucket(s) you selected!
⬇️ Download the .csv file - you'll upload it to Pluto in the next step.
Create an User / Access Key in AWS IAM
In Pluto, create an experiment and continue to the assay data upload step. Select the Other option and follow the link the grant Pluto access to the bucket. You'll upload the credentials file that you downloaded in the previous step, and receive an email notification when all of the assay data files have been imported.
After uploading the credentials file, you can click Skip or Annotate your samples on the screen above to move to the sample annotation step while your assay data is being imported.