Over the past week, we have been hearing about the horrible natural disaster headed our way called “Hurricane Irene”, preparing to hit New York City as a Category 2 or 3 major storm. Oh, scratch that, make it a Category 1. Wait, it hit today at 9:00AM as a Tropical Storm. Here in upstate New York, we never did expect much more than tropical storm winds, maybe with hurricane gusts. What no one warned us about was how the rain was we were getting would create catastrophic flooding! In any case, the plentiful rain and even gale force winds were likely to create power outages, or at least power blips. For the supercomputing center I work at, power blips can be the worst menace.
Power blips cause the chillers to go offline, but most of the computers, particularly the Blue Gene/L, continue to run and produce heat. The data center can get very hot very quickly – which is very bad for computers. Additionally, our roof can start to leak, once you get about half an inch of rain. We were expecting 4-8 inches! We are likely to get leaks in places we haven’t seen them before.
Given the likelihood of power blips, it was a fairly easy decision to shut down the computers for the duration of the storm, and my colleague Adam and I were able to complete the shut down in a few minutes time (leaving a few systems running that are either critical, or in danger of disk sticktion). But because of the concern of the roof leaking, we wanted to protect some of our computers from water damage – in particular, we needed to protect the Blue Gene/L, which is no longer manufactured and is effectively irreplaceable.
We had some plastic sheeting to cover the Blue Gene/L rows, but getting it to stay in place was no easy task. The Blue Gene/L is cooled by cold air under the raised data center floor blowing into the system, being expelled at the top where it is exhausted by an air intake vent. This air flow continues even with the Blue Gene/L turned off. Plastic sheeting balloons up and off the top.
So of course we tried to tape it down. But there is a lot of force, and the tape is ripped right off.
So of course we tried to weigh it down, but with what? Data center floor tiles came to mind, but while looking for a cart to collect some unused ones, Adam noticed a pallet of steel plates. These seemed heavy, and we tried those.
Remember when I said there was a lot of force lifting that plastic? Those iron plates crept over the edge until Adam noticed that one of them was about to brain me. Clearly this was not a very safe solution, and probably would not hold until the storm passed anyway.
While return the steel plates to the pallet, Adam noticed a couple of folding tables in the storage closet. Two might hold the plastic on one Blue Gene/L row! But while trying to get them out, we upset a container of glass Christmas ornaments. After cleaning up that mess, we found that there were no more folding tables around anyway.
What about the original plan of floor tiles? We put some plastic on one row, and a couple of tiles held it down! But the middle was still billowing. A door from a computer rack put that down. We tried to repeat this with the second row – that time, a tile slid on to me, leaving some minor damage. After washing up, and a great deal more effort, we got the second row covered.
After that, we put tiles in position first, slid in the plastic, and added a door. The third and fourth row went very fast! Since the edges were billowing out a bit, we taped them down. A little “caution” tape, and we were done.
We will see Monday how the center faired. Hopefully the plastic was not really needed. I do suspect the power has “blipped” at least once.