MSI’s shared scratch storage volume has been heavily utilized during the months of June and July, so much so that on one occasion all jobs writing to scratch failed because scratch reached its capacity. Since this incident, we have been working one-on-one with users to prevent future failures by moving some data off of scratch to other MSI storage resources better suited for longer-term data storage. While one-on-one engagements have helped, starting August 1 we plan to implement a couple new measures to safeguard against scratch filling and causing jobs writing to scratch to crash.
Here are the steps we plan to take:
- If scratch utilization reaches 85%, the MSI Helpdesk will contact the top 25 users of scratch to ask them to reduce their scratch storage footprint.
- If scratch utilization reaches 93%, data 15 days and older will be moved to the data owner's group home directory, starting with those users using the most scratch storage space.
If data is moved from scratch to a group's home directory, an email will be sent to the data owner and to the persons who administer the PI group account, so that members of this group can take whatever action is needed to manage the group's quota.
Please feel free to reach out to us if you have questions about this change. MSI recently deployed new storage resources to help groups manage large data volumes over the long and short term. We would be happy to set up a time to discuss your research data requirements and what resources we maintain that could be used to help manage these requirements. Contact email@example.com for assistance.