I work in Kalispell, Montana, as an anesthesiologist and head of our group of 20 anesthesiologists. We have two adjoining hospitals, one that functions as an outpatient surgery center, the other an acute care hospital and trauma center. We recently had an incident involving our Mindray anesthesia machines and monitors. Six (6) Mindray A5 anesthesia machines and eight (8) DPM 7 patient monitoring systems were in use delivering anesthesia care at the time of the incident. We have both Mindray anesthesia machines and a few Fabius Gas machines. All of our monitoring is Mindray.
While one of the Mindray machines was being moved from one anesthetizing location to another, a network connection was made incorrectly by an anesthesia technician, who plugged both ends of the network cable into the network receptacle rather than one end into the network receptacle and the other end to the anesthesia machine. This network was installed for the exclusive use of the Mindray equipment, thus no other equipment was affected. This misconnection resulted in a loop where the network traffic consumed 100% of the bandwidth. When the misconnection occurred, no other Mindray anesthesia machines or patient monitors were able to communicate with the network. As a result, every Mindray machine and patient monitor in both buildings simultaneously shut off and refused to turn back on as long as they were connected to the network. It was discovered that if the machine was disconnected from the network, the machine and monitor returned to normal function. Word spread to all anesthesiologists to unplug the machines from the network and, within 15 to 20 minutes, everything was back in service, but disconnected from the network. Fortunately, there were no untoward sequela for the patients, but every patient had their anesthetic, monitoring, and mode of ventilation changed. Hospital information technology found the source of the excessive network traffic and broke the loop by unplugging the offending cable from the network. The network operation was returned to normal within a few hours.
Carl Tinlin D.O. Kalispell, MT
Mindray Reply:
On May 19, 2015, Mindray was notified of a situation involving six (6) A5 and eight (8) DPM 7 systems which unexpectedly shutdown; the result of a cabling error where a Kalispell anesthesia technician inadvertently connected a cat5 cable from one wall connection to another wall connection, forming a loop on the Kalispell-installed and maintained hospital network. Mindray determined the cause of the shutdown was due to overwhelming network broadcast traffic. The issue was resolved by disconnecting the Ethernet to EMR cable. The systems then restarted and functioned normally. No patient injury was reported.
The customer requested an investigation as to why Mindray’s devices were unable to withstand the problem caused by the user error. The results of the investigation are as follows:
- Mindray was initially unable to reproduce the issue using the specific Kalispell LAN settings provided, but upon further examination of Kalispell’s network topology, Mindray was able to recommended specific LAN switch settings that would disable any switch port that received network broadcast or multicast traffic at a rate that would cause Mindray’s A5 or DPM 7 systems to shut down.
- Mindray Engineering and Kalispell Biomedical Engineering concurrently but independently determined that turning off the Cisco port-fast feature would enhance the solution.
- When reproducing the identified issue in the Mindray lab, it was observed that while the A5 User Interface did go blank, the ventilator continued to ventilate as intentionally designed.
We appreciate the collaborative nature of the effort put forth by the Kalispell staff to work with Mindray’s engineering team. Through this process we have identified the cause of the shutdown and made recommendations for Kalispell’s network. Additionally, Mindray will make software enhancements to strengthen the network interface and will continue to develop and incorporate, where possible, future product and software enhancements to provide additional protection against unanticipated broadcast or multicast traffic, as our goals include providing the safest and most reliable products possible.
Rich Cipolli Vice President, Product Development Mindray, North America Mahwah, New Jersey
The APSF Committee on Technology The information provided is for safety-related educational purposes only, and does not constitute medical or legal advice. Individual or group responses are only commentary, provided for purposes of education or discussion, and are neither statements of advice nor the opinions of APSF. It is not the intention of APSF to provide specific medical or legal advice or to endorse any specific views or recommendations in response to the inquiries posted. In no event shall APSF be responsible or liable, directly or indirectly, for any damage or loss caused or alleged to be caused by or in connection with the reliance on any such information.