An introduction to the challenges and options in moving scientific data over the network.
Learning Outcomes
You understand the challenges and options in moving scientific data over the network
Demonstrate understanding of advantages and disadvantages of various bulk data transfer tools.
Understand appropriate solutions for a data transfer workflow, taking into account current computing, storage and network infrastructure, and end-point sites.
Demonstrate the ability to identify, communicate, and mitigate potential bottlenecks in collaboration with campus cyberinfrastructure and network operators.
Transfer data using Rclone.
Transfer data using Globus.
Readings
Preparation
A few things to check prior to this workshop.
Install RClone
Installing RClone on your local machine
Scientific Data Transfer Examples
Be able to identify real world examples of data transfer issues that can be fixed.
File Transfers with Remote Computers
Understand how to transfer files using wget, scp, and rsync.
Experiential Learning
1. Ice Breaker
Data Movement Ice Breaker
2. Introduction to Scientific Data Networks
Understand how networks connect everything and how UH is connected.
3. Networks
Understand what networks, and the equipment that connects everything, look like.
4. Data Transfer Evaluation of the Network
Understand what tools we use to test network throughput.
5. Processes and Queues
Understand how data actually moves between machines and explain queues and buffers.
6. Transmission Control Protocol (TCP)
Understand what transmission control protocol is.
7. Transfer Programs
Be able to identify common/best transfer applications.
8. Globus
Understand what Globus is.
9. Globus Installation
Understand how to setup and use Globus to move data.
10. Transferring Data using Globus
Understand how to setup and use Globus to move data.
11. Configuring and Using Rclone
Understand how to configure and use Rclone.
12. Transferring Files with Rclone
Understand how to transfer files using Rclone.
Assessments
Help us assess this workshop
Provide feedback to the workshop organizers
Outcome(s) assessed:
You understand the challenges and options in moving scientific data over the network