How to transfer data from one storage system to another

Before the transfer

• Ensure that you only transfer necessary data. Remove unnecessary files such as unused programs, temporary files, and duplicates to prevent filling up available space during the transfer process.
• Consider the suitable location for your data:
   
  • Note that the destination must be suitable for the data to be transferred and that you must have access rights to the destination. Check the storage quota of the destination and the data classification in advance.
  • Important work files and research data are recommended to be stored on the network drive in the group directory. Order storage space from Aalto's data center file services if your project does not already have it: https://selfservice.esupport.aalto.fi/ssc/app#/order/2088/?cid=40537
• Allocate sufficient time for the transfer. Be prepared to conduct the transfer on campus or via a virtual desktop infrastructure (VDI). If the transfer spans multiple days, ensure your computer remains on and connected to Aalto University's network. It is advisable to carry out the transfer over a weekend. Additionally, verify your computer settings to ensure it stays on throughout the process.
• Consider the transfer method and ensure there is enough available space on your computer if the transfer method uses your computer’s disk space. For example, during file downloads, the files are usually first stored in a temporary folder on your computer (often the ‘Temp’ folder). Some tools, such as rsync, rclone or azcopy (in Azure), are designed to optimize the transfer process.
  
  If you are transferring large amount of data with Windows, use VDI DataTransferMachine. Instructions for transferring data via a virtual desktop: Data transfer from OneDrive via VDI | Aalto University
• Consider compressing files before transferring if necessary. Compression is a good option if the amount of data is large and compression significantly reduces the file size. Compressed files take up less space during the transfer, which can speed up the transfer and reduce bandwidth usage. However, compressed data creates temporary files that often need to be decompressed at the destination, which temporarily requires as much free storage space as the uncompressed data.
  o   Describe the contents of the package separately so that its contents are clear before downloading.
  o    Zip format and tar + gz in Linux are the most common compression formats. 7zip (.7z) is the most efficient but requires the same software to open the package.

When starting the transfer

• Transfer only a small number of files at a time to avoid failure due to large size and to prevent excessive load on the available disk space. It is recommended to start with smaller sections, for example, not more than 100 GB.
• Use a wired network connection and ensure a fast network connection to both the source and the destination. Preferably, use general-purpose servers or a virtual desktop infrastructure (VDI). Do not transfer over Wi-Fi or via remote connection (VPN) from outside the university!
• Do not perform other memory-intensive tasks on the computer at the same time.
• Allocate enough time for the transfer and perform it during a quiet time to ensure that the transfer process is not interrupted. Note that you may need to leave your computer in place during the transfer to prevent interruptions to the network connection.

After the transfer

• Verify that the files work correctly in the new location. The presence of the files in the new location does not yet ensure that they work correctly.
• Remove the data from the old location as soon as possible after verification.
• Note that storage systems often have backup and versioning settings that retain and use the original storage quota even after data has been deleted. For example, on network drives, ‘snapshot’ backups are retained for 8 weeks.

Help from IT Service Desk

If you are transferring data from one network drive to another, you can request help from IT Service Desk (servicedesk@aalto.fi). We are happy to help. 

Provide the following details for the transfer:

• From where and to where the data needs to be transferred. Provide the exact directory paths.
• The most suitable time for the transfer; provide a few options. The time must be agreed upon in advance.
• Whether there is a suitable new folder at the destination or if a temporary folder needs to be created. A temporary transfer folder might be necessary if the data in the old location does not follow the same structure as in the new location and needs to be modified. The temporary folder should be deleted within the agreed timeframe, for example, by the end of the year.

Related information

• There are services for transferring and sharing large files, such as Funet FileSender and SFTP (read more on the Sharing large files page on aalto.fi). Both methods, however, require downloading the files to the local disk during the transfer process.