As the Internet titans struggle to find more cost effective methods of
keeping their datacenters cool, while simultaneously lowering their impact on the environment, I can’t help but smile. No I’m not insensitive the impending crisis that is global warming – to the contrary, I am a big believer in
thinking globally and acting locally. It’s just that those of us in the field of BSM have already been executing on this global-local philosophy for years:sweeping, expensive measures aren’t always the best remedy – sometimes you just need to improve how you use what you already have.
Several years ago I was asked to provide some consulting services to a customer, an unnamed manufacturer, with a plant in Canada that desperately needed a solution to a simple but a very expensive problem. The servers in the datacenter that hosted the
technology controlling the operation of the assembly line had to maintain a temperature that did not vary above or below a (+/-) 5 degree Celsius. If it did the servers would automatically shut themselves down to prevent overheating. With a vehicle rolling off the production line every 27 seconds, any stoppages on the manufacturing line had a clear and measurable impact on revenue.
Some proposals called for grand
schemes to overhaul the datacenter with sweeping and sophisticated new solutions…until I arrived and offered a simple answer: integrate the tools you already have to develop an early warning system to monitor the temperature in the datacenter.
The HVAC systems already installed in the datacenter had onboard SNMP servers, which allowed a periodic polling of the datacenter temperature at three different points. The outcome of this poll was correlated to the required mean temperature in the datacenter – and a
severity level for the production floor then conveyed to the Center. For example, if the temperature varied
more than 2 degrees Celsius either way, IT operations staff was automatically paged and the manager on duty received an email on his blackberry. In just a week after installing this very simple early warning system, the number of production line stops was reduced by over 90 percent.
This situation was perfect for a BSM solution which is designed to integrate and correlate IT and business data and visualize it in a way that is meaningful to the business. More importantly, it exemplifies what is often symptomatic of a larger problem: providing a way for IT to understand that given current course and speed, a crash will occur in X-number of minutes. Whether it’s global warming or cooling the datacenter, sometimes we already have the ends of a solution in our hands – we just need to tie them together.
-- Jonathan
Ugggghhh. I would have never thought I would have seen those two words together. Let's not jump on the buzz word bandwagon!
I think our clients need to first figure out how to solve the organizational problems and get the facilities groups and the IT groups aligned. I haven't come across many Data Center Facilities Engineers in my day who know much about BSM and how their PDU's and CRAC's support the business. If we start with the organizational challenge (put a DC Facilities Engineer on-shift 24x7 with IT Ops), then integrate the facility and building management systems (FMS/BMS) into the broader IT management systems, we can start talking about real alignment and value.
The real study would be looking into what Google is doing here!
Doug
BSM/ITSM Blog: http://dougmcclure.net
Posted by: Doug McClure | April 17, 2008 4:54 AM
I agree with Doug's comment, organizational problems are the biggest obstacle.
Engineers in commercial hosting facilities tend to have better access to the thermostat control. Enterprise offices with their own mini data centers (comms rooms) have a different problem. Comms rooms are usually very local and the environmental monitoring is provided by the building's FM team.
Not long ago I worked somewhere where the FM team handed off monitoring overnight to the security guard on duty that night. The problem there was that security guards tend to go walkabout in order to complete their nightly rounds. If an environmental alarm were to light up on their station they did not see it until they returned from their rounds. Meanwhile, on the other side of town we had three shift operators in the data center bridge dedicated to alarm watching. So we did the same, integrated a feed from the comms room environmental sensors into Managed Objects so that our data center team could alert the building FM team within seconds as opposed to relying on the security guard who may have fallen asleep.
The challenge for us though was not the technical integration, it was establishing the correct escalation processes and procedures for the environmental alerts. When the FM team realized we could offer this notification service they wanted to take advantage of our services to overcome technical limitations of the enivronmental sensors. The sensors had simple thresholds whereas our BSM solution could implement complex logic such as waiting for a period of time before escalating the alarm or being aware of planned maintenance in the comms room which may alter the levels of heating or cooling.
- Mosh
Posted by: Mosh Jahan | May 10, 2008 1:02 PM