City of Baltimore Without Power

It wasn’t the whole City of Baltimore, but the downtown area where a lot of the government offices are located, including city hall. I used to work in the Mayor’s Office of Information Technology and know some of the IT setup down there. One of the two main server rooms has no generator or UPS, so, I’m sure it went dark real quick. A lot of critical infrastructure was in that building, including the mail gateway and the main mail server. I heard they had geo-clustering in place so the mail server could survive this kind of outage (the 2nd server is in the building with the generator) but that it wasn’t working properly. Not a surprise, after evaluating the geo-cluster product, we decided that it would probably cause more problems than it solved. The crazy part is that I specifically recall meetings about power planning for the building without the generator, those included installing a generator and or installing a large UPS unit. That was over two years ago. I guess it didn’t happen.

What’s neat is that I am in the process of designing a highly redundant Exchange environment where I work now. Not that we don’t have generators, but in the event of a disaster that would render one of our two primary datacenters inoperable, we will be able to restore services and data within four hours. It is all about what level of service your business is willing to pay for. I’m thinking that the IT guys down at the City of Baltimore were counting on luck more than they were courting on solid design. I doubt that “all day” is an acceptable downtime window. Typically, it isn’t the IT worker’s bad design, it is a general lack of understanding of the technology and its limitations by the management there. I found myself thinking “if only they would listen”. I’m sure the IT team there is thinking the same thing. There are probably a few “I told you so’s” too.