You are here
Primary Storage (Panasas) Migration Guide
On January 9, 2013, MSI systems (excluding Koronis) moved to a new primary storage system to provide high speed, high capacity storage for home directories and project space. The structure of home directories and project space has changed and will require many users to migrate files and change scripts and programs. Details of the changes as well as instructions to assist with migration tasks are below.
MSI home directories will be unified across all systems except Koronis. All files will be located in /home/group_name/username. Files in the 'default' home (this is the home directory used on login.msi.umn.edu, nx.msi.umn.edu, lab machines, the lab batch queue, and isub) will be transferred automatically; no user action is necessary. Files on Itasca, Cascade and Calhoun will NOT be transferred. Since files may share names but have distinct content, merging these directories automatically could cause files to be lost.
|System(s)||Old Location||New Location||Files transferred automatically?|
|default (login.msi, nx.msi, labs, isub)||/home/msi/username||/home/group/username||Yes|
Files that are not automatically transferred can be accessed (Read Only) in the Old Location listed above for a period of 6 months.
Symbolic links will be maintained that will direct /home/msi/username to /home/group/username. This means that any references to /home/msi/username in scripts and programs will still work correctly for the "default" home. The canonical way to refer to home directories, however, will be /home/group/username.
The new storage is now in place and functioning, as of January 2013. The new storage will outperform /project and can be mounted on HPC compute nodes. The new storage allows jobs to be run directly from shared space, which is not possible from the current /project. Data for inactive groups was deleted from /project space on July 15, 2013. For all active groups, the existing project space (/project) will be unavailable after September 15, 2013. Users must migrate any needed files to their new space at /home/group .
Project space on the new system is available at /home/group/shared. This directory is group readable; anyone in the group can browse the group's shared directory.
Intergroup sharing can be accomplished by placing files in a public space located at /home/group/public. This directory is world readable; any MSI user can browse a group's public directory.
You can learn more about your storage options at the Storage webpage.
Due to the compression of files on the existing project space, files copied to the new storage might require more disk space. If you reach the limits of your quota, you can delete unwanted files, use the tar utility to reduce the number of files and amount of disk space used, or request or purchase additional space. You can find more details about purchasing additional storage on the website at the Dedicated Storage Space webpage.
MSI is also providing the capability of tape storage at cost. You can find more details about the cost for tapes on the website at the Dedicated Storage Space webpage.
Per-user and project-space quotas will no longer be enforced on the new system. Instead, all files in /home/group will be counted against a per-group quota. Quota increases will need to be made by the PI or group admin of a group as they affect group-wide allocations. The initial quota of a group will be the sum of all user quotas and any existing project space quota.
Also see our FAQ regarding checking quota on the new system.
Windows homes (the U: drive) are slated to be migrated to the new system on a yet to be decided date. Details and relevant instructions will be added when available.
Note that if you do not use Itasca, Calhoun, Cascade, or project space, you do not need to do anything. The migration will be automatic.
Files will remain available in their old locations for a period of 6 months. Please ensure that you have migrated your files within that time.
- Files on the new storage system take up a minimum of 32 KB. This means if you have a large number of small files, space utilization may be significantly more on the new system. You can use the 'tar' command to collect small files into one, thus reducing space consumption. For example, to collect all files in a directory called 'smallfiledir' into an archive called 'archive_name.tar.gz', use the following command:
tar czf archive_name.tar.gz smallfiledir/
- Files on the current /project are compressed by the system. The new system will not compress files. Depending on the type of files and the degree to which they were compressed, you may see notable file size inflation in the transfer.
Home directories on the HPC systems will not have files transferred automatically. It will therefore be necessary to move any files desired from the old location to the new location. There are several strategies than can be used to bring in files from HPC homes, each described below.
If you're not sure whether your home was on it1/it2 or xe1/xe2, try both; if you choose incorrectly the command will fail harmlessly with a "No such file or directory" error.
The below procedures should be followed on either login.msi.umn.edu or the system you will be transferring files for. For example, to move your itasca home, login to either itasca.msi.umn.edu or login.msi.umn.edu.
Create a subdirectory
This is the simplest way, as it copies all files without needing to handle any conflicts. The example below moves the contents of an itasca home directory into a new directory called 'itascahome':
mkdir ~/itascahome rsync -av /home/it1/user/ ~/itascahome
If you have only a few directories or files you wish to keep from your HPC home, you can simply copy those over, assuming there is no file or directory with the same name already in your home. The below commands will overwrite any such existing files or directories; ensure that the name is not already in use before copying.
You can use the 'cp' command to copy individual files:
cp /home/it1/user/file ~/
The rsync command is recommend for copying directories, as it is restartable and more flexible than cp. If your rsync transfer is interrupted, simply run the command again; files already transferred will be skipped.
rsync -avz /home/xe2/user/directory ~/
Attempt to merge the directories
You may attempt to completely merge your home directories. Be aware, however, that it will be impossible to cleanly merge; there will be files that share names in both the old home and the new. Only one can exist in the new home.
The following command will merge an Itasca home with the new home, skipping files that already exist in the new home:
rsync --ignore-existing -avz /home/it2/user/ ~/
You may wish to replace a file in your new home with one from an Itasca or Calhoun home. This can be done with the commands above.
Transfers from old /project should take place on login.msi.umn.edu.
As there are no files automatically transferred into the new project space, you can move the entire contents of the old space to the new with a command such as the one below:
rsync -avz /project/group/ /home/group/shared
Be aware, however, of the following caveats:
- /project is not a high performance storage system. If you have especially large amounts of data, a transfer may take some time to complete.
- If there are files in /project that are not group readable and you are not the owner, you will receive "Permission denied" errors when attempting to copy these files. Either the owner of the file can perform the transfer or change the permissions, or the PI or group admin can request that MSI staff move the files and/or change their permissions.
- You may have links to project space. These will need to be redirected. You can find any links you have to project space from MSI Linux systems with the commands:
cd find . -type l -lname "/project*" -ls
- See the above additional caveats.
Individual files and directories can also be transferred on an 'as needed' basis if desired.
Any scripts (such as a PBS script) that refer to the absolute location of an HPC home (e.g. /home/it1/username/file) will no longer function, as the path to the home directory will have changed. Scripts that refer relatively, (e.g. ~/file or $HOME/file) will refer to the correct location, however the files themselves still must be copied from the old location.
For example, suppose you have a PBS script as below:
#!/bin/bash -l #PBS -l walltime=01:00:00,pmem=2500mb,nodes=1:ppn=8 cd /home/it1/user/job ./task.pl > out.dat
Such a script will no longer work, since the old home location (/home/it1/user) is no longer writable (and will not be available at all after 6 months).
Suppose you have moved all of your Itasca scripts into a directory called 'itascahome' in your new home directory. The following command could be used to replace all instances of '/home/it1/user' with '/home/group/user/itascahome' in a file called 'filename', allowing that file to function correctly again:
sed -i -e 's#/home/it1/user#/home/group/user/itascahome#g' filename
You may have a directory full of scripts, all of which need paths fixed. The command below will search the current directory and replace '/home/it1/user' with '/home/group/user/itascahome' in all plain text files (compiled programs will be skipped):
grep -l --binary-files=without-match '/home/it1/user' * | xargs sed -i -e 's#/home/it1/user#/home/group/user/itascahome#g'
As with absolute paths in scripts, any symbolic links that refer absolutely to a home directory or a project space will need updating. Many users have not made use of symbolic links and will not need to address this problem. If however, you attempt to access symbolic links and receive "No such file or directory" errors, you may need to update the link.
The following command can be used to identify any links in the current directory whose target begins with '/home/xe2/user':
find . -lname '/home/xe2/user*'
This could be done similarly for links containing a reference to /project.
A single link could be fixed manually as follows (for example):
ln -s -f /home/group/user/calhounhome/file linkname
All links discovered by 'find' could be fixed with code as follows:
for link in $(find . -lname '/home/xe2/user*') do readlink $link | sed 's#/home/xe2/user#/home/group/user/calhounhome#' | xargs -I target ln -f -s target $link done
Many compiled programs include absolute paths in makefiles as well as binaries. It may be necessary to recompile these programs. To remove any old files from previous compilation, most programs use the standard makefile target "clean". To invoke this target, issue the following command in the program's directory:
Also be aware that binaries compiled on one system (such as Itasca's SB nodes) will not necessarily run on any other systems. For this reason, it is advisable to separate programs that are particular to one system.
Files such as .bashrc and .vimrc, which control configuration of various programs, will use the version contained in the default home. This may cause unexpected behavior in certain cases, such as when a .bashrc attempts to load a module not available on every system. Problematic commands may need to be removed in such cases. Refer to the Software FAQ for details on maintaining one .bashrc file for all the different MSI systems.