El Centro Regional Medical Center‘s Director of IS John Gaede intends to use innovative medical technologies to transform healthcare. “We can attack escalation in healthcare cost by using IT to avoid test duplication, improve communication, and make it easier for physicians, nurses, and clinicians get their work done.”
“We’ve already put in an Electronic Health Care Record system and an operating room nurse charting system. Now we’re bringing in Vocera [for hands-free voice] and wireless IV pumps for bedside drug administration. Soon, all new stat orders will go directly to Symbol wireless handheld devices, and we’ll have the ability to shoot medical images over wireless,” said Gaede.
“We had all of these great new technologies coming at us, but we didn’t have a way to support them. We realized that the direction we wanted to go as an organization was going to require a really robust wireless foundation—a state of the art wireless network.”
But creating the network that ECRMC needed, at a price this 165-bed acute care facility could afford, was no small challenge. After a five-vendor bake-off, Gaede ripped out his aging Proxim Orinoco WLAN and deployed an extensible cooperative mesh from Aerohive Networks. “We’re up to about 60 HiveAPs now. When you roll out something like this, you just know whether or not you made a good decision—and I still think we made the best decision possible.”
Spotting the symptoms
Two years ago, ECRMC had several new healthcare applications coming on-line that demanded reliable, secure wireless roaming. At the time, the hospital’s WLAN consisted of 40 autonomous Proxim Orinoco APs, mounted throughout the facility.
“Those APs were great back in their day, but they’re now eight years old,” explained Gaede. “We got a lot out of the money that we invested in them, but we were rapidly catching up in terms of healthcare IT. When we decided to bring up our operating room nurse system for paperless charting, we knew there was no way we could do that with Orinoco.”
In fact, WLAN usage had already flat-lined due to issues related to seamless roaming—or lack thereof. “In healthcare, it’s a big deal when you can [access and edit] full patient records, live at the bedside. But when we went live with our state-of-the art Electronic Health Care Record system, those applications would just drop dead,” said Gaede.
Specifically, when wireless users roamed beyond a single AP’s footprint, they found themselves disconnected (at least briefly), requiring PC reboot to bring EHCR applications back up. “Our staff really had to roam to get their work done. When physicians just stopped doing any bedside entry that was a red flag—the catalyst for us to say: We can no longer use this WLAN.”
Diagnosing the disease
Nonetheless, before embarking upon a potentially costly rip-and-replace, ECRMC had to be absolutely certain it was choosing the right next-generation wireless architecture and platform.
“We’d already put in one system where we had overlooked the need for very reliable wireless,” said Gaede. “I have to stand before the board and tell them that if they spend money on this, we’ll be successful. So I spent weeks talking to vendors—I wanted to really understand why they did what they did.” Gaede also spoke with CIOs at many other medical centers, small and large, in the US and abroad, quizzing them about their own selection criteria and post-deployment experiences.
The requirements list that emerged set the center’s bar pretty high. “We wanted something that would be highly available, highly reliable, and have good performance. Even if we have to start with 802.11a/g APs [to reduce initial cost], we wanted the ability to easily step up to 802.11n because getting near-desktop speed for physicians when placing orders is very important to us.”
“We also have many places in the hospital with old thick concrete walls, so we needed a platform with depth,” explained Gaede. This not only meant RF technology that could accommodate and overcome attenuation and interference—it also demanded a wireless backhaul strategy to avoid Ethernet drops in hard-to-wire venues.
The center’s next WLAN would also need to support a diverse mix of demanding applications, from latency-sensitive wireless VoIP handsets and badges to bandwidth-intensive medical imaging applications. “We needed quality of service controls,” said Gaede. “And of course we also had to comply with HIPAA.”
But ECRMC still had to implement this ready-for-primetime WLAN on a rural medical center’s modest budget. In particular, Gaede took a hard look at the resources the hospital could afford to put into WLAN management and troubleshooting. “I’m not going to get another FTE for wireless—we have ten people on our entire IT staff,” he said. “Ease of deployment and implementation were essential; we needed something that we could figure out on our own.”
Prescribing the cure
After doing his homework, Gaede had narrowed his candidate list down to five: Aerohive, Aruba, Cisco, Extricom, and Meru. To address its roaming and RF challenges, the hospital initially focused on single-channel architectures.
“Extricom came in first,” said Gaede. “I spent about six weeks with them, but they could not make their system work here. It failed over and over—connecting and disconnecting. Finally, they sent an Extricom engineer who was able to fix the problem, but at that point we’d had it.”
Gaede issued his next on-site test invitation to Meru. Their tests started well but didn’t end that way. “We took them down to our emergency room because we knew that was a tough environment. But we’d watch signal strength on a laptop go from 5 to 3, 5 to 1, and then our applications would just drop dead down there.”
Disheartened, Gaede decided to reconsider multi-channel architectures. “Cisco came back with a [price quote] that was 75% higher than anyone else, so we eliminated them. We then conducted [an in-situ] bake-off between Meru, Aruba, and Aerohive.”
Gaede and his staff started with the hospital’s emergency room application, MEDHOST. They asked each vendor to prove that MEDHOST could run without disruption on a wheeled cart as it moved throughout the hospital. Brief reachability losses were detected by a continuous ping, but the test’s objective was to verify seamless AP-to-AP roaming with sufficiently short hand-off delay.
Next, Gaede’s team tested resilience, using the emergency department as an RF-hostile testbed. “There are many competing [radio transmissions] down there, including cordless phones and imaging equipment,” explained Gaede. “Meru brought in external antennas to overcome that interference, but ended up creating multiple cells using different channels anyway.” This test eliminated single-channel architectures, narrowing the bake-off to Aruba and Aerohive.
Tests continued in other parts of the hospital where connections were known to be difficult, looking for stability and non-stop charting when moving into and beyond locations like the ER. In the end, ECRMC selected Aerohive over Aruba based on ease of deployment and technology.
“Aruba matched Aerohive’s price, so why should I take a chance and go with this newer company that doesn’t have an established healthcare footprint? After careful consideration, I believed that Aerohive’s intelligent APs and cooperative control were the superior technology,” said Gaede.
“I also felt that we wouldn’t be a little fish in a big pond if we went with Aerohive. I needed a real partner so that when we do have a problem—when I deploy that new wireless medical device that nobody else is using yet—[my WLAN vendor] is going to be there for us.”
Nursing the network back to health
According to Gaede, the hospital’s “wireless guru” spent less than an hour studying Aerohive documentation to configure and install the center’s first HiveAP. Another administrator then used that global policy to deploy 56 more 802.11a/g HiveAPs and four 802.11n HiveAPs. All were installed without any on-site assistance from Aerohive.
With Aerohive’s cooperative architecture, policies are created on and disseminated by a HiveManager. The HiveManager is not a WLAN controller—it simply serves as a central repository for configuration and monitoring data. As each new HiveAP is connected to the network, it seeks out the HiveManager to initialize itself and “go live” without administrator assistance.
But even plug-and-play configuration cannot answer that essential question: Where should APs be placed for optimum coverage and performance? For the sake of simplicity, the hospital started with a one-for-one swap, replacing old Proxim APs with new HiveAPs. “As we did that, we felt that our coverage density was far more than we needed, so we started to skip spots,” said Gaede.
Those extra HiveAPs were used to move beyond the hospital’s existing Proxim footprint, supporting wireless patient registration at four outpatient clinics operated by the medical center. “We’re planning to bridge over subnets [using GRE] to give our physicians the freedom to roam from clinic to hospital, and we’re also starting to [cover] areas of the hospital campus that we didn’t before.”
Another factor in the center’s relatively kink-free rollout has been Aerohive’s cooperative mesh architecture. HiveAPs have been able to relay traffic to each other between floors inside the hospital and in outpatient clinics where no Ethernet drops previously existed. Because medical applications raise the stakes on “mission critical,” ECRMC depends upon Aerohive’s self-healing adaptive mesh and controller-free architecture to avoid single-point-of-failure.
Pumped about the future
Although Gaede is pleased with performance to date, this new WLAN has yet to be subjected to heavy loads. “In areas that are heavily using [our WLAN], we now have about 15 devices per AP. At the low end, in the OR we have four APs and 24 laptops. But that’ll change dramatically when we put Vocera, wireless IV pumps, and bedside phlebotomy in this spring,” he said.
“My ER is now using wireless and they love it. We had so much confidence by that point that we put APs in our OR just two days before we went live on a new paperless system in there. We had no blips, no errors, no issues at all—that speaks volumes,” said Gaede.
“Our pharmacists have thanked us for putting up the new wireless network, because now they can take a laptop with them as they move through the campus to process their medicine orders. We just received Aerohive’s Guest Manager, which we’re planning to roll out to give visitors a 24-hour pass to get onto a VLAN to use the Internet. We’re going to have Aerohive come out to look at what we’ve done to make sure that we’re prepped to roll out Vocera this February.”
In short, Gaede is pleased with progress to date and excited about the future. With this robust foundation now in place, ECRMC can stop plugging leaks in the plumbing and truly focus on delivering better medicine through more creative use of innovative wireless applications.
Lisa Phifer owns Core Competence, a consulting firm focused on business use of emerging network and security technologies. A 27-year industry veteran, Lisa has been involved in Wi-Fi training, product evaluation, network deployment, and security assessment since 2001.
This article was first published on WiFiPlanet.com.