Data Received - E5 Coral WGBS from Genohub Project 2216545 Using Rclone on Owl

2025
E5
coral
WGBS
BS-seq
bisulfite sequencing
Genohub
2216545
Data Received
rclone
Author

Sam White

Published

February 3, 2025

INTRO

Whole genome bisulfite sequencing (WGBS) data was received from Genohub for Project 2216545, as part of urol-e5/deep-dive-expression. Genohub provided Amazon AWS S3 credentials for downloading. Installation of the AWS command line interface (CLI) was a bit too involved to try to figure out on Owl (our Synology server), which might be BSD Linux? Instead, I installed and configured rclone on Owl.

METHODS

Configuring Rclone

Installation was a breeze. SSH’d into and then used the following command provided on the rclone Downloads page:

sudo -v ; curl https://rclone.org/install.sh | sudo bash

I performed the initial configuration step using the credentials supplied by Genohub.

  • rclone config

Then, I ran the list command to view the contents of the S3 bucket on AWS:

rclone ls remote:

However, this did not work!

2025/02/03 12:57:43 NOTICE: Failed to ls: operation error S3: ListBuckets, https response error StatusCode: 403, RequestID: SH6FBZZSFZVEFBMG, HostID: <redacted>, api error AccessDenied: User: arn:aws:iam::<redacted>:user/remote is not authorized to perform: s3:ListAllMyBuckets because no identity-based policy allows the s3:ListAllMyBuckets action

After some sleuthing in the Genohub documentation for using the AWS CLI, I realized I needed to specify the S3 bucket. In order to specify an S3 bucket with rclone, you need to create an rclone alias - you cannot configure S3 to access a bucket directory.

So, I set up an alias with rclone:

  1. Run rclone config

  2. Specify n) New remote

  3. name> genohub

  4. Specify alias as storage type:

    Type of storage to configure.
Choose a number from below, or type in your own value

>alias

  5. remote> remote:genohub_bucket

Note

You need to replace <genohub_bucket> with the actual bucket ID provided by Genohub.

  1. Save the configuration at the prompt.

Then, you can actually access your data using the alias genohub:

rclone ls genohub:

DATA

All files were downloaded to owl:/volume1/web/nightingales/E5-coral-deep-dive-expression/genohub2216545 and MD5 checksums verified:

Note

An astute reader might notice that the screenshot below shows a different directory than that listed above. At the time of downloading, I wasn’t sure which project this belonged to. I have since updated and moved the data to the correct directory.

Screenshot of MD5 checksum verifications.