7. (Our $0.02 about) Transfer Programs
Overview
Questions
- What are some of the most common/best transfer applications?
Objectives
- Introduce some of the basic data transfer applications used in research.
Scp, sftp, rsync, other SSH-based
- Very accessible
- Have a problem with long-haul
- https://www.psc.edu/hpn-ssh-home/hpn-ssh-faq/
- Can be patched to work much better, if you are invested in an ssh-based workflow
- Common tools in this category are single-threaded
- Versus support for parallel transfers
GridFTP (including Globus GridFTP)
- Breaks files into small pieces, transfers many pieces concurrently, and reassembles them at the receiving end
- Tends to get better performance out of networks with symptoms.
- May not tend to get better performance out of transfer paths with other problems
- Authentication and licensing might present obstacles to use in ad-hoc workflows
Lftp
- Very versatile (Russian origin) program with features such as mirroring and parallel transfers
Key Points
- SFTP, although common, is not the best application for moving data and there exists other SSH versions and tools that are better to transfer large data.