11. Using Rclone

Rclone is a free utility for syncing directories between object storage systems (such as Amazon S3, Dropbox, Google Drive etc) and file based storage (e.g. home or scratch))

https://rclone.org

Rclone in KOA

Rclone is installed on KOA and can be used in the command line via

$ rclone

Configuring Rclone

Before you can use Rclone, you must configure it This configuration step will set up access for the remote object storage system that you want to transfer data to and from In this tutorial we will configure Google Drive since UH has Google for Education and everyone at UH has it

Open a Shell Session on KOA

1.) Start a shell session on KOA through your own terminal or you can use Open OnDemand via [https://koa.its.hawaii.edu/] *You WILL be prompted for DUO two-factor verification.

2.) Navigate to Clusters -> _Koa_Shell_Access, to start your session

Configuring Rclone

$ rclone config

$ n

No remotes found - make a new one

n) New remote

s) Set configuration password

q) Quit config

Choose a name for the remote object storage system

You’ll be prompted for the name of the remote object storage system\, we use “rclone-gdrive” in this tutorial

$ rclone-gdrive

Choosing a Storage Option

$ 18

• Choose #18, Google Drive as Storage Option ** See help for drive backend at: https://rclone.org/drive/ **

Google Application Client Id

*Setting your own is recommended, see https://rclone.org/drive/#making-your-own-client-id for how to create your own.

• If you leave this blank, it will use an internal key which is low performance.
• I leave this blank in my tutorial

Client Secret

• I also leave this blank in my tutorial

OAuth Client Secret

• Leave blank normally.

Scope

• 1 Full access all files, excluding Application Data Folder. “drive”

• 2 Read-only access to file metadata and file contents. “drive.readonly” Access to files created by rclone only.
• 3 These are visible in the drive website. File authorization is revoked when the user deauthorizes the app. “drive.file” Allows read and write access to the Application Data folder.
• 4 This is not visible in the drive website. “drive.appfolder” Allows read-only access to file metadata but
• 5 does not allow any access to read or download file content. “drive.metadata.readonly”

$ 1

Leave ID of the root folder blank normally

Advanced Configuration

$ n

Auto Authenticate

In this tutorial we are running RClone on a remote machine (KOA).

$ n

# Optional config_token

# Access the config_token from your local RClone
1.) Confirm RClone is successfully installed on your machine by running rclone version in your CLI of choice.
2.) From the KOA shell copy/paste and run the rclone authorize command into your local CLI

$  rclone authorize "drive" "<AUTH CODE>"

Example: Windows Powershell

3.) A browser window should open, prompting you to sign into your Google Drive Account

4.) Allow RClone to access your Google Drive

5.) Once you allow access, you will see a ‘Success’ screen:

6.) Navigate back to your CLI, to view the generated token, copy the token

Example: Windows Powershell

7.) Paste the token into KOA

$  config_token> <YOUR CONFIG TOKEN>

Configuring of Google Drive

-Do not configure as a team drive

$ n

$ y

• Quit Configuration

$ q