Data center developer and operator 365 Main Inc. has issued detailed information regarding the root cause behind why back-up power generators in the company's San Francisco facility failed to start during a PG&E; power outage last week, resulting in approximately 40 percent of customers in the facility losing power to their equipment for up to 45 minutes.
According to the statement, at 1:47 p.m. on Tuesday, July 24, 365 Main's San Francisco data center was impacted by a power surge caused when transformer breakers at a local PG&E; power station unexpectedly opened. PG&E; has still not determined what caused the breakers to open. Typically when a power outage occurs, the outage triggers 365 Main's rigorously maintained and tested back-up diesel generators to start-up and take over providing power supply to customers. 365 Main's San Francisco facility has ten 2.1MW back-up generators to be used in the event of a loss of utility power. Eight primary generators can successfully power the building, with two generators available on stand-by in case there are any failures with the primary eight. However, following the power outage last week, three of 365 Main's 10 back-up power generators, manufactured by Hitec, failed to complete their start sequence. A complete investigation of the incident began immediately.
Within hours of the incident, an international team of specialists was deployed to 365 Main's San Francisco data center facility to join on-site technicians and begin systematically testing the generators in search of a root cause. After days of thorough testing around the clock, the team discovered a weakness in an essential component of the back-up generator system known as a DDEC (Detroit Diesel Electronic Controller).
The team discovered a setting in the DDEC that was not allowing the component to correctly reset its memory. Erroneous data left in the DDEC's memory subsequently caused misfiring or engine start failures when the generators were called on to start during the power outage on July 24.
The investigation team discovered DDEC issues on each of the failed Hitec units and were able to successfully simulate failure. A fix was introduced by altering the timing of a command to the DDEC component, allowing more time between the engine shut-down command and the DDEC reset command. Once this fix was introduced, the Hitec generators successfully passed more than 50 consecutive start-up sequence tests without incident.
The testing methodology was performed by Hitec specialists along with 365 Main's chief technician and staff. Specialists from Cupertino Electric were present during all testing, and EYP Mission Critical Facilities will provide independent verification of the findings the week commencing 6 August. 365 Main has implemented the DDEC fix in its San Francisco and El Segundo facilities. Of the five data centers in 365 Main's portfolio, the San Francisco and El Segundo facilities are the only ones with Hitec generators containing DDECs. All other facilities feature other brands of generators or have different models of Hitecs.
365 Main is sharing the discoveries of its investigation with other Hitec customers. In addition, Hitec has expanded its preventative maintenance procedures as a direct result of discoveries made during the 365 Main investigation.
In the wake of the outage, 365 Main published an apology to customers and daily updates directly from the investigation team meeting minutes, allowing customers and the public at large to track progress. A complete archive of these updates and more details about today's update are available at: www.365main.com/status_update.html
Chris Dolan, president and CEO of 365 Main, said: "365 Main has a track record of providing customers with data centers that are considered to be among the world's finest. We extend our sincere apologies to customers who were impacted by this incident. Addressing customer concerns is our top priority. In the days since the incident occurred, we have identified and corrected the root source of the problem and are taking steps to prevent this type of problem from happening again. We are also making our comprehensive findings available to other data centers to try to prevent the same problem from recurring elsewhere.
Glenn Ellis, president and CEO of Hitec USA, also commented: "Our top priority is taking steps to prevent this type of unforeseen incident from occurring again. We sincerely apologize to 365 Main and its customers that our generators failed to deliver the continuous power as designed."
As part of their service level agreements with 365 Main, 365 Main customers receive rent abatements (refunds) in the event that electrical power is dropped in the section(s) of the data center where their servers are located. 365 Main has said it is honoring all service level agreements with affected customers.