Acquiring new facilities will come with challenges
Data Center facility owners often obtain new sites through mergers and acquisitions of other data center companies. This comes with many challenges, including bringing the newly acquired facilities’ mechanical and electrical systems up to the same operational standards they have set for their other data center locations within their portfolio. To meet the needs of the clients these facilities typically need upgrades and replacements to the mechanical and electrical infrastructure. To perform these upgrades the facility owner first requires a detailed account of the current system. This could include original construction drawings of the structural, mechanical, electrical, plumbing and/or one-lines of these systems. There also needs to be a level of operational knowledge among facility staff to reduce human error in upgrading and operating these complex systems.
Human error can occur at any step in the development process
BUT, there are critical measures the facility owners can take to ensure that little to no human error will impact the facility’s improvements. A Mission Critical, SiTESPAN client in the northeast exemplifies this scenario. The client had recently acquired two new facilities to their portfolio and they needed to bring those facilities up to their operating standards. These standards are not only required internally but are also set by their clients needs. SiTESPAN received a phone call on a Sunday detailing the mission critical client’s needs and by Monday SiTESPAN was on site. SiTESPAN staff began development of the implementation Method of Procedures (MOPs) for one of the new facilities’ infrastructure upgrades. This meant developing the MOP documentation that outlined step by step procedures with check points and verifications to help prevent unnecessary impact to IT load.
Knowledge Transfer among staff is key
Two major problems were presented immediately. The first was the lack of knowledge transfer between facility staff. In the week prior, the Critical Facilities Manager hired a new manager for the facility and he then unexpectedly left the organization. A significant client of the colocation owner became concerned with the change in staff as the owner was planning a new critical power infrastructure install in the coming weeks. This install included new PDUs and busways which required startup and commissioning with load banking. With the new electrical equipment, testing is needed before going live to ensure proper installation and operation. For example, there is a risk in energizing PDUs due to the inrush current, usually seen during the first several power cycles of transformer energization, and these short but significant amounts of current can cause circuit breakers to trip and equipment to shut down.
Facility information your staff should know to help reduce facility failures that are a result from lack of knowledge transfer:
Make sure all relevant site documentation and location is passed on (Drawings, O&M, maintenance & testing, MOP/SOPs, warranty info, contracts, maintenance schedules, one-lines
Discuss equipment history
Master Plan of any future upgrades
Current facility challenges (EOL equipment, failed/nonoperational equipment, lack of capacity)
Facility and systems design and capacity (i.e. withstand 100mph winds, seismic rated, 2N UPS, N+2 generator…)
Second Major Problem Found … Human Error
This led to the second major problem that was uncovered by SiTESPAN which contained a litany of human errors. First, the owner did not have current, as-built documentation for the facility. The most recent power distribution one-lines were not up to date and contained erroneous information. SiTESPAN also discovered that several of the existing critical path circuit breakers appeared to have been set at unexpected levels with some found at their factory default. Multiple firms had previously completed multiple coordination studies, presumably as part of power infrastructure additions, but the settings on many of the circuit breakers did not match the recommended settings within the studies. The conflicting coordination studies resulted in an intensive field study conducted by SiTESPAN. This study covered roughly 425 critical infrastructure breakers and over half of the breakers documented required setting adjustments. Equipment labeling was also lacking, and in some cases mislabeled. Modifications had been made to the power distribution system, but the labeling of the gear was not updated to reflect the changes. Many pieces of equipment had identifiers hand-written in pencil or temporarily labeled.
Human Errors can be avoided when proper protocols are in place. Here are a few suggestions to help reduce risk of human error in your technical facility:
Follow proper safety protocols
Have reviewed and approved MOP/SOP’s for needed activities
Proper tools, test equipment, ppe
Buddy system – Two people working together for checks and balances
Trained Staff (safety, mop/sop’s, equipment specific, site specific)
Work with up to date facility documentation
Creating the Method of Procedures under a deadline
The MOPs for the upcoming infrastructure additions needed to be completed quickly. SiTESPAN’s focus was initially on the parts of the power distribution system that would be impacted by the infrastructure additions. SiTESPAN first developed the necessary one-line diagrams and implemented MOPs; and labeling and breaker setting changes to the infrastructure upgrade equipment. The focus then turned to a more holistic approach to the entirety of both facilities. After a significant amount of field verification by the seasoned operations and engineering personnel, one-line diagrams and operational MOPs were created. Labeling modifications for consistency throughout the site were also recommended. SiTESPAN completed a full and current coordination study for both facilities.
The diagram above depicts an non-finalized construction one-line that was created prior to the construction start date of the client’s project. SiTESPAN found the client’s site to be drastically different than what was recorded, for example the dashed areas are to represent the newly installed equipment but instead it is outlining where current/original equipment sits.
SiTESPAN developed the above one-line for the client based on the site coordination study. The diagram was updated by accurately defining the As-Found system through color coded power rails, validating equipment size & load, removing unnecessary nomenclature and correcting the configuration.
Developing & Managing an execution plan
Implementation of the recommended circuit breaker settings is currently ongoing. Because there are varied circumstances surrounding breaker setting modifications, the changes cannot be made entirely at once in a live data center. To manage this effort, SiTESPAN developed an execution plan where circuit breaker setting changes could be coordinated, prioritized and scheduled to minimize risk and client impact. With so many data center outages being attributed to human error during operations, these facilities were in jeopardy of failures due to lack of up-to-date information, procedures and site conditions. SiTESPAN’s mission critical knowledge and ability to provide the clients with this level of service allows for data center operational success.
A Full Detailed list of site services offered and performed by SiTESPAN include:
- Performed coordination studies for two facilities
- Assistance with coordination study implementation and documentation
- ARC Flash Analysis
- Created one-line diagrams with coordinating labeling recommendations
- Created detailed MOPs for both construction and operations
- Provided on-site support for operation of the data centers
- Performed troubleshooting to solve multiple, existing operational issues
- Conducted thorough site investigation and verification of conditions
- Interviewed staff and corrected or improved operational conditions
- Recommended expectations in the vendors’ processes for installation and maintenance