Primary Storage (Panasas) Migration Guide

On January 9, 2013, MSI systems (excluding Koronis) moved to a new primary storage system to provide high speed, high capacity storage for home directories and project space. The structure of home directories and project space has changed and will require many users to migrate files and change scripts and programs. Details of the changes as well as instructions to assist with migration tasks are below.

 

What Will Change

MSI home directories will be unified across all systems except Koronis. All files will be located in /home/group_name/username. Files in the 'default' home (this is the home directory used on login.msi.umn.edu, nx.msi.umn.edu, lab machines, the lab batch queue, and isub) will be transferred automatically; no user action is necessary. Files on Itasca, Cascade and Calhoun will NOT be transferred. Since files may share names but have distinct content, merging these directories automatically could cause files to be lost.

Linux home directories

System(s) Old Location New Location Files transferred automatically?
default (login.msi, nx.msi, labs, isub) /home/msi/username /home/group/username Yes
Itasca, Cascade /home/it1/username
OR
/home/it2/username
/home/group/username NO
Calhoun /home/xe1/username
OR
/home/xe2/username
/home/group/username NO
Koronis /home/koronis/username No Change N/A

Files that are not automatically transferred can be accessed (Read Only) in the Old Location listed above for a period of 6 months.

Symbolic links will be maintained that will direct /home/msi/username to /home/group/username. This means that any references to /home/msi/username in scripts and programs will still work correctly for the "default" home. The canonical way to refer to home directories, however, will be /home/group/username.

Project Space

The new storage is now in place and functioning, as of January 2013. The new storage will outperform /project and can be mounted on HPC compute nodes. The new storage allows jobs to be run directly from shared space, which is not possible from the current /project. Data for inactive groups was deleted from /project space on July 15, 2013. For all active groups, the existing project space (/project) was shut down on September 15, 2013.

Project space on the new system is available at /home/group/shared. This directory is group readable; anyone in the group can browse the group's shared directory.

Intergroup sharing can be accomplished by placing files in a public space located at /home/group/public. This directory is world readable; any MSI user can browse a group's public directory.

You can learn more about your storage options at the Storage webpage.

MSI is also providing the capability of tape storage at cost. You can find more details about the cost for tapes on the website at the Dedicated Storage Space webpage.

Quotas

Per-user and project-space quotas will no longer be enforced on the new system. Instead, all files in /home/group will be counted against a per-group quota. Quota increases will need to be made by the PI or group admin of a group as they affect group-wide allocations. The initial quota of a group will be the sum of all user quotas and any existing project space quota.

Also see our FAQ regarding checking quota on the new system.

Windows home directories

Windows homes (the U: drive) are slated to be migrated to the new system on a yet to be decided date. Details and relevant instructions will be added when available.

Migration Instructions

Note that if you do not use Itasca, Calhoun, Cascade, or project space, you do not need to do anything. The migration will be automatic.

Files will remain available in their old locations for a period of 6 months. Please ensure that you have migrated your files within that time.

Caveats

  1. Files on the new storage system take up a minimum of 32 KB. This means if you have a large number of small files, space utilization may be significantly more on the new system. You can use the 'tar' command to collect small files into one, thus reducing space consumption. For example, to collect all files in a directory called 'smallfiledir' into an archive called 'archive_name.tar.gz', use the following command:
tar czf archive_name.tar.gz smallfiledir/
  1. Files on the current /project are compressed by the system. The new system will not compress files. Depending on the type of files and the degree to which they were compressed, you may see notable file size inflation in the transfer.

Transfer files from HPC homes (Itasca, Calhoun, Cascade)

Home directories on the HPC systems will not have files transferred automatically. It will therefore be necessary to move any files desired from the old location to the new location. There are several strategies than can be used to bring in files from HPC homes, each described below.

If you're not sure whether your home was on it1/it2 or xe1/xe2, try both; if you choose incorrectly the command will fail harmlessly with a "No such file or directory" error.

The below procedures should be followed on either login.msi.umn.edu or the system you will be transferring files for. For example, to move your itasca home, login to either itasca.msi.umn.edu or login.msi.umn.edu.

Create a subdirectory

This is the simplest way, as it copies all files without needing to handle any conflicts. The example below moves the contents of an itasca home directory into a new directory called 'itascahome':

mkdir ~/itascahome
rsync -av /home/it1/user/ ~/itascahome 

Copy files and directories individually

If you have only a few directories or files you wish to keep from your HPC home, you can simply copy those over, assuming there is no file or directory with the same name already in your home. The below commands will overwrite any such existing files or directories; ensure that the name is not already in use before copying.

You can use the 'cp' command to copy individual files:

cp /home/it1/user/file ~/

The rsync command is recommend for copying directories, as it is restartable and more flexible than cp. If your rsync transfer is interrupted, simply run the command again; files already transferred will be skipped.

rsync -avz /home/xe2/user/directory ~/

Attempt to merge the directories

You may attempt to completely merge your home directories. Be aware, however, that it will be impossible to cleanly merge; there will be files that share names in both the old home and the new. Only one can exist in the new home.

The following command will merge an Itasca home with the new home, skipping files that already exist in the new home:

rsync --ignore-existing -avz /home/it2/user/ ~/

You may wish to replace a file in your new home with one from an Itasca or Calhoun home. This can be done with the commands above.

Transfer files from /project

On September 15, 2013 MSI’s old project space (/project) was taken off-line. On July 1, 2014 all data will be removed from the disk. If there is any data that you wish to preserve, please contact MSI help, help@msi.umn.edu, before the July 1 deadline.

Fix absolute paths in scripts

Any scripts (such as a PBS script) that refer to the absolute location of an HPC home (e.g. /home/it1/username/file) will no longer function, as the path to the home directory will have changed. Scripts that refer relatively, (e.g. ~/file or $HOME/file) will refer to the correct location, however the files themselves still must be copied from the old location.

For example, suppose you have a PBS script as below:

 

#!/bin/bash -l
#PBS -l walltime=01:00:00,pmem=2500mb,nodes=1:ppn=8

cd /home/it1/user/job

./task.pl > out.dat

 

Such a script will no longer work, since the old home location (/home/it1/user) is no longer writable (and will not be available at all after 6 months).

Suppose you have moved all of your Itasca scripts into a directory called 'itascahome' in your new home directory. The following command could be used to replace all instances of '/home/it1/user' with '/home/group/user/itascahome' in a file called 'filename', allowing that file to function correctly again:

sed -i -e 's#/home/it1/user#/home/group/user/itascahome#g' filename

You may have a directory full of scripts, all of which need paths fixed. The command below will search the current directory and replace '/home/it1/user' with '/home/group/user/itascahome' in all plain text files (compiled programs will be skipped):

grep -l --binary-files=without-match '/home/it1/user' * | xargs sed -i -e 's#/home/it1/user#/home/group/user/itascahome#g'

Fix symbolic links

As with absolute paths in scripts, any symbolic links that refer absolutely to a home directory or a project space will need updating. Many users have not made use of symbolic links and will not need to address this problem. If however, you attempt to access symbolic links and receive "No such file or directory" errors, you may need to update the link.

The following command can be used to identify any links in the current directory whose target begins with '/home/xe2/user':

find . -lname '/home/xe2/user*'

This could be done similarly for links containing a reference to /project.

A single link could be fixed manually as follows (for example):

ln -s -f /home/group/user/calhounhome/file linkname

All links discovered by 'find' could be fixed with code as follows:

for link in $(find . -lname '/home/xe2/user*')
do
  readlink $link | sed 's#/home/xe2/user#/home/group/user/calhounhome#' | xargs -I target ln -f -s target $link
done

Compiled software

Many compiled programs include absolute paths in makefiles as well as binaries. It may be necessary to recompile these programs. To remove any old files from previous compilation, most programs use the standard makefile target "clean". To invoke this target, issue the following command in the program's directory:

make clean

Also be aware that binaries compiled on one system (such as Itasca's SB nodes) will not necessarily run on any other systems. For this reason, it is advisable to separate programs that are particular to one system.

Configuration files

Files such as .bashrc and .vimrc, which control configuration of various programs, will use the version contained in the default home. This may cause unexpected behavior in certain cases, such as when a .bashrc attempts to load a module not available on every system. Problematic commands may need to be removed in such cases. Refer to the Software FAQ for details on maintaining one .bashrc file for all the different MSI systems.