High-Throughput Data Transfer for Astrophysics Simulations


MSI, with the support of the University of Minnesota’s NTS (Network and Telecommunication Services), has recently deployed a system to take advantage of the ten-gigabit network known as the Northern Lights GigaPoP network. With this system in place, MSI was able to help Peter J. Mendygral, a post-doctoral member of MSI principal investigator Tom Jones’s research group and employee of Cray, with some important work. Pete needed to transfer about seven terabytes of data in less than two days. He was able to do this successfully by utilizing MSI’s high-performance TeraScala storage system to store his remote data and using Globus GridFTP software to effect the transfer. Pete sat down with a MSI staffer to talk about his research, experience with MSI, and how MSI resources have helped him succeed. 


MSI: What type of research are you doing?


Pete: My research is in astrophysics. I study magnetohydrodynamics, which means fluid dynamics simulations of ionized plasma with magnetic fields and their effects. I also study what we understand to exist in galaxy clusters, which are the largest gravitationally bound systems in the universe. So, specifically what I look at are the interactions between outflows of supermassive black holes that we believe exist at the centers of some of the large galaxies in these galaxy clusters. Within these galaxies are black holes that emit jets, supersonic jets. On a very large scale, they can have an enormous influence on the evolution of the clusters as well as the individual galaxies and stars.


MSI: What resources previously and currently have you utilized here at MSI?


Pete: Primarily we’ve run these simulations of jets and galaxy clusters on Itasca. I’ve also focused on other more specific phenomena, like the build-up of magnetic fields in galaxy clusters and the creation and evolution of cosmic ray particles. We have also done some smaller, kind of one-off type of things mainly for verifying the codes that I have written. Collaborators and I have done simulations of solar wind, galactic super bubbles, and giant bubbles created in galaxies by supernovas. My primary focus has been on the outflows of black holes and galaxy clusters, in terms of the simulations I have run on MSI systems.  In addition, I use the LMVL systems a lot for my movie creations. In the LMVL I do a lot of stereo renderings because it really gives me an edge in understanding what’s going on inside a simulation. Seeing that third dimension is critical for understanding these simulations.


MSI: How has your use of the Northern Lights GigaPoP network in coordination with MSI systems and our high-performance TeraScala storage system enhanced your capabilities as a researcher?


Pete: I did my simulation this particular time on the Kraken system [the image above is from that simulation] at the National Institute for Computational Science (NISC) in Tennessee and it produced around seven terabytes of data. It’s often taken for granted in high-performance computing that you can produce your calculation relatively quickly, but then you come across the problem of dealing with this enormous dataset, especially when you run your simulation at a non-local site. So getting it in-house for analysis and creating movies become difficult challenges. The traditional route would be to send a secured copy via remote access (i.e. the ‘scp’ command), but this process would have taken at least 90 days. In this particular instance I had a conference coming up in a couple of weeks and I needed the data as soon as possible.  So, 90 days was not going to work for me. The other option was to try to coordinate someone on the other end filling up a large number of hard drives and manually mailing them to me.  That would require a large personnel cost and, best-case scenario, it would have still taken about two weeks. So what MSI’s Jeff McDonald and David Porter did for me was set up an MSI hardware interface with the Northern Lights GigaPoP network and I was able to move the entire dataset in less than two days. Because of this I was able to start working on my data immediately. This is an absolutely critical ability to have.


With the ever-changing needs of our users, MSI is proud to be able to help meet their challenges and to support researchers like Pete Mendygral in his pursuit of cutting edge research.


See all Research Spotlights.