The world is in the midst of a data revolution. In response the five universities that form the MGHPCC consortium have each initiated a new data science institute, initiative or program, but how and where to store all that big data? The Northeast Storage Exchange (NESE) is a shared regional storage resource funded by the National Science Foundation which aims to do just that.

NESE is housed at the Massachusetts Green High Performance Computing Center (MGHPCC) and operated and maintained as a long-term partnership between the MGHPCC, its university partners Boston University (BU), Harvard University, the Massachusetts Institute of Technology (MIT), Northeastern University (NEU), the University of Massachusetts (UMASS) network, and Red Hat Inc.
In particular, NESE is expected to be a boon for those who have need of large volume (multi PB), high performance or archival storage with high bandwidth connectivity to the national research networks, specifically the flood of data expected from the inexorable advances in computing and storage technology, arising, for instance, through the "internet of things" or advances in DNA sequencing speed. “Over time, we will provide essentially all useful types of storage from POSIX file systems to block store for Cloud infrastructures to mounts to Globus endpoints and long-term archives,” says project Principal Investigator Saul Youssef, a Research Associate Professor in the Department of Physics at Boston University.

Youssef cites four keys strengths underpinning NESE’s success:

First its location at MGHPCC, placing it only a few feet from five of the great research universities in the Northeast. “It's hard to imagine a better spot,” says Youssef. “Such close shared proximity means that we can easily have high-speed networking with low latency to five institutions at once.”

Second, the agreement of Scott Yockel's FAS Research Computing team at Harvard (rc.fas.harvard.edu) to take on building and operating NESE for the long term. “Getting Scott and his deep and talented 25 person team’s commitment to NESE is a very significant indicator of long term success,” says Youssef.

Third, Youssef points to NESE’s relationship with Red Hat Inc. “Because we want to be offering multiple different storage types with a commonly managed back end, Ceph (https://ceph.com/ceph-storage/) storage was a natural choice for us. Through a major agreement between BU and Red Hat Inc. (https://www.bu.edu/rhcollab), we have had major contributions and ongoing involvement via Red Hat, including software licenses and the direct involvement of Sage Weil and his Ceph team within Red Hat.”

Finally, Youssef cites basic economics. “For research projects in the region, NESE storage is going to be far less expensive than either commercial cloud storage or running your own storage,” says Youssef. “For computing that's already happening at the MGHPCC, it is also going to be quite easy to get high bandwidths to NESE storage. This means that if we can do a good job technically, NESE will be in a position to grow rapidly with buy-ins from research, engineering and education projects, not to mention possible partnerships with local technology companies, biotechs and, startups.”