Skip to content

Via Rclone

Rclone is a powerful, open-source command-line program designed for managing files across cloud storage platforms. Often described as the "Swiss Army knife" of cloud storage management, Rclone allows users to sync, copy, and transfer files between various cloud storage providers including Google Drive, Dropbox, Amazon S3, Microsoft OneDrive, and many others. Since its creation in 2012 by Nick Craig-Wood, Rclone has become an essential tool for system administrators, developers, and tech-savvy individuals who need to efficiently handle data across multiple cloud environments. With its robust encryption capabilities, extensive configuration options, and ability to handle files of virtually any size, Rclone provides a versatile solution for both personal and enterprise cloud storage management needs.

Reference instructions for mounting on Windows by SURF can be found here These steps overlap with the instructions below. Other instructions for interacting with the reseach drive using Rclone are here.

Installation

Manual Download and Install

  • Download and install Rclone from Rclone's website
  • Verify installation by running rclone --version in your terminal

When using Windows, you can use the rclone command only from the folder where rclone.exe is located. The executable can be moved if needed.

Script Download and Install

To install rclone on Linux/macOS/BSD systems, run:

sudo -v ; curl https://rclone.org/install.sh | sudo bash

Generate App Credentials

  • Go to Research Drive Settings > Security
  • Click "Create new app password"
  • Note down the password

Obscure Password

  • Open your terminal
  • Run the command: 
    rclone obscure YOUR_APP_PASSWORD
  • Copy the outputted obscured password string

Get WebDAV URL

  • Go to Research Drive > File settings (in bottom left corner) > WebDAV
  • Copy the WebDAV URL

Configuration Setup

Create or edit the rclone configuration file: - Linux/MacOS: ~/.config/rclone/rclone.conf - Windows: %USERPROFILE%\.config\rclone\rclone.conf

Add the following configuration: 

[cropxr]
type = webdav
url = YOUR_WEBDAV_URL
vendor = nextcloud
user = YOUR_APP_USERNAME 
bearer_token_command = exit
pass = YOUR_OBSCURED_PASSWORD

Replace: - YOUR_APP_USERNAME with your email address that you use to login to Research Drive - YOUR_OBSCURED_PASSWORD with the obscured password from step 3 - 'YOUR_WEBDAV_URL' with the URL from step 4

If you have previously setup this configuration with owncloud, the password needs to be regenerated from within Research Drive, and the password, url and the vendor need to be updated in the config.

Mounting Commands

Linux/MacOS: 

rclone mount "cropxr:research_drive/folder" /local/folder/ --vfs-cache-mode writes --use-cookies -v

Windows:

For mounting in Windows, you need to install WinFsp. This means that you need administative priviledge to install programs on your computer, or you need to ask the systems admins for this program in advance. In the terminal from the folder that contains rclone:

rclone mount "cropxr:research_drive/folder" K: --vfs-cache-mode writes --use-cookies -v

The --vfs-cache-mode writes flag is an important Rclone configuration option that controls how file data is cached when using Rclone to mount a remote file system. When set to "writes", this option means:

  • Files opened for reading are not cached initially
  • Files opened for writing are cached during the entire upload
  • File data read from the remote while a file is open is cached

The --use-cookies flag in Rclone enables the use of cookies for authentication and session management when connecting to remote servers, particularly WebDAV servers like Research Drive (which is based on OwnCloud).

Replace the paths in the example command with the actual path needed. The path of the folder on the research drive, should be either the complete path, or the path that is visible to/shared with the user. This likely looks like this for the full path cropxr:cropxr (Projectfolder)/investigations/investigation_folder/study_folder or like this for shared folders cropxr:study_folder

Best practice is to mount a specific folder (e.g., WPC1) rather than the root of Research Drive. This helps in managing permissions and access control more effectively.

Copy Commands

rclone copy /my/source/location "cropxr:research_drive/destination/folder"
Replace the paths in the example command with the actual path needed. The path of the folder on the research drive, should be either the complete path, or the path that is visible to/shared with the user. This likely looks like this for the full path cropxr:cropxr (Projectfolder)/investigations/investigation_folder/study_folder or like this for shared folders cropxr:study_folder

This copies the file from the source to the research drive. When the source location is a folder, all the contents of the folder are copied to the desination. For large files it is recommended to add a time-out. The timeout should be 10 min per GB of the largest file that will be copied.

rclone copy --timeout 50m /my/source/location "cropxr:research_drive/destination/folder"